# EMB / comps > Source: Benchize (https://benchize.com) — https://benchize.com/benchmarks/vals-ai%3Aemb%3Acomps. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/benchmarks/vals-ai%3Aemb%3Acomps. Evaluating agents on Excel-based financial modeling tasks Canonical URL: https://benchize.com/benchmarks/vals-ai%3Aemb%3Acomps Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Evaluating agents on Excel-based financial modeling tasks Professional work · Models tested: 47 · Score publishers: Vals AI ## Model results | # | Model | Score | ± | Cost | Latency | | | --- | --- | --- | --- | --- | --- | --- | | 1 | [Gemini 4 Argon](https://benchize.com/models/gemini-4-argon-high.md)High | 82.98% | SE ±3.49 pp | $11.77 | 2,301.06source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.979 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 11.770794 \| USD \| per_test \| Lower is better \| \| `latency` \| 2301.06 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 262144 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.49 pp sourcePopulation 76 stderr 3.492 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-4-argon Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-4-argon` | | | | | | | | 2 | [Claude Fable 5.1](https://benchize.com/models/claude-fable-5-1-max.md)Max | 82.23% | SE ±4.05 pp | $24.6 | 2,943source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.233 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 24.604393 \| USD \| per_test \| Lower is better \| \| `latency` \| 2942.999 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.05 pp sourcePopulation 76 stderr 4.046 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5-1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-fable-5-1` | | | | | | | | 3 | [Claude Sonnet 5.5](https://benchize.com/models/claude-sonnet-5-5-max.md)Max | 80.04% | SE ±4.26 pp | $10.25 | 2,463.28source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.042 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.248719 \| USD \| per_test \| Lower is better \| \| `latency` \| 2463.278 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.26 pp sourcePopulation 76 stderr 4.256 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-sonnet-5-5` | | | | | | | | 4 | [Muse Wonder](https://benchize.com/models/muse-wonder-max.md)Max | 79.49% | SE ±3.84 pp | $6.19 | 996.37source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 79.488 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.189429 \| USD \| per_test \| Lower is better \| \| `latency` \| 996.365 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.84 pp sourcePopulation 76 stderr 3.841 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-wonder Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/meta~1muse-wonder` | | | | | | | | 5 | [GPT-6 Astra](https://benchize.com/models/gpt-6-astra-max.md)Max | 77.95% | SE ±3.66 pp | $12.62 | 1,532source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.953 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.624116 \| USD \| per_test \| Lower is better \| \| `latency` \| 1532.002 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.66 pp sourcePopulation 76 stderr 3.655 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-astra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/openai~1gpt-6-astra` | | | | | | | | 6 | [Claude Haiku 5.5](https://benchize.com/models/claude-haiku-5-5-max.md)Max | 77.93% | SE ±4.26 pp | $1.77 | 1,326.93source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.925 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.766809 \| USD \| per_test \| Lower is better \| \| `latency` \| 1326.925 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.26 pp sourcePopulation 76 stderr 4.262 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-haiku-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-haiku-5-5` | | | | | | | | 7 | [Gemini 3.8 Flash](https://benchize.com/models/gemini-3-8-flash-high.md)High | 76.44% | SE ±4.03 pp | $9.1 | 667.12source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 76.435 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 9.096536 \| USD \| per_test \| Lower is better \| \| `latency` \| 667.116 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.03 pp sourcePopulation 76 stderr 4.025 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.8-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3.8-flash` | | | | | | | | 8 | [Claude Opus 5.5](https://benchize.com/models/claude-opus-5-5-max.md)Max | 76.41% | SE ±6.14 pp | $20.51 | 4,242.93source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 76.405 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 20.510975 \| USD \| per_test \| Lower is better \| \| `latency` \| 4242.927 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.14 pp sourcePopulation 76 stderr 6.136 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-opus-5-5` | | | | | | | | 9 | [Claude Opus 5](https://benchize.com/models/claude-opus-5-max.md)Max | 75.17% | SE ±4.51 pp | $7.37 | 1,435.22source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.171 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.365041 \| USD \| per_test \| Lower is better \| \| `latency` \| 1435.222 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.51 pp sourcePopulation 76 stderr 4.512 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-opus-5` | | | | | | | | 10 | [Gemini 3.7 Flash](https://benchize.com/models/gemini-3-7-flash-high.md)High | 74.61% | SE ±4.17 pp | $7.44 | 773.41source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 74.609 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.439465 \| USD \| per_test \| Lower is better \| \| `latency` \| 773.405 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.17 pp sourcePopulation 76 stderr 4.166 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.7-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3.7-flash` | | | | | | | | 11 | [GPT-6 Sol](https://benchize.com/models/gpt-6-sol-max.md)Max | 73.3% | SE ±6.3 pp | $1.73 | 562.43source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.295 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.730675 \| USD \| per_test \| Lower is better \| \| `latency` \| 562.426 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.3 pp sourcePopulation 76 stderr 6.296 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/openai~1gpt-6-sol` | | | | | | | | 12 | [Claude Opus 4.8](https://benchize.com/models/claude-opus-4-8-max.md)Max | 72.8% | SE ±4.2 pp | $12.89 | 2,442.71source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.797 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.886468 \| USD \| per_test \| Lower is better \| \| `latency` \| 2442.712 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.2 pp sourcePopulation 76 stderr 4.198 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-8 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-opus-4-8` | | | | | | | | 13 | [GPT-5.6 Sol](https://benchize.com/models/gpt-5-6-sol-max.md)Max | 71.31% | SE ±6.2 pp | $9.61 | 814.32source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 71.309 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 9.611009 \| USD \| per_test \| Lower is better \| \| `latency` \| 814.317 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.2 pp sourcePopulation 76 stderr 6.203 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/openai~1gpt-5.6-sol` | | | | | | | | 14 | [Claude Fable 5](https://benchize.com/models/claude-fable-5-max.md)Max | 70.79% | SE ±3.98 pp | $16.06 | 1,608.16source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.785 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 16.057068 \| USD \| per_test \| Lower is better \| \| `latency` \| 1608.159 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.98 pp sourcePopulation 76 stderr 3.984 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-fable-5` | | | | | | | | 15 | [GPT-6 Luna](https://benchize.com/models/gpt-6-luna-max.md)Max | 70.47% | SE ±6.16 pp | $0.37 | 1,820.73source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.474 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.373356 \| USD \| per_test \| Lower is better \| \| `latency` \| 1820.73 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.16 pp sourcePopulation 76 stderr 6.155 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/openai~1gpt-6-luna` | | | | | | | | 16 | [Gemini 3.6 Flash](https://benchize.com/models/gemini-3-6-flash-high.md)High | 70.02% | SE ±3.84 pp | $4.73 | 603.57source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.019 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.730378 \| USD \| per_test \| Lower is better \| \| `latency` \| 603.567 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.84 pp sourcePopulation 76 stderr 3.84 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3.6-flash` | | | | | | | | 17 | [Gemini 3.5 Flash](https://benchize.com/models/gemini-3-5-flash-high.md)High | 69.79% | SE ±4.04 pp | $8.1 | 1,282.56source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.788 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.099855 \| USD \| per_test \| Lower is better \| \| `latency` \| 1282.564 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.04 pp sourcePopulation 76 stderr 4.035 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3.5-flash` | | | | | | | | 18 | [MiMo-V2.6-Flash](https://benchize.com/models/mimo-v2-6-flash.md) | 69.62% | SE ±5.92 pp | $0.11 | 1,230.32source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.617 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.110268 \| USD \| per_test \| Lower is better \| \| `latency` \| 1230.32 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.92 pp sourcePopulation 76 stderr 5.916 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/xiaomi~1mimo-v2.6-flash` | | | | | | | | 19 | [Claude Sonnet 5](https://benchize.com/models/claude-sonnet-5-max.md)Max | 69.38% | SE ±6.42 pp | $12.92 | 3,392.35source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.384 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.922728 \| USD \| per_test \| Lower is better \| \| `latency` \| 3392.345 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.42 pp sourcePopulation 76 stderr 6.423 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-sonnet-5` | | | | | | | | 20 | [MiMo-V2.6-Pro](https://benchize.com/models/mimo-v2-6-pro.md) | 68.26% | SE ±6.46 pp | $0.37 | 2,820.67source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.264 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.373493 \| USD \| per_test \| Lower is better \| \| `latency` \| 2820.674 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.46 pp sourcePopulation 76 stderr 6.462 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/xiaomi~1mimo-v2.6-pro` | | | | | | | | 21 | [Muse Spark 1.3](https://benchize.com/models/muse-spark-1-3-xhigh.md)Extra high | 64.8% | SE ±5.67 pp | $4.91 | 1,365.92source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.798 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.913012 \| USD \| per_test \| Lower is better \| \| `latency` \| 1365.924 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.67 pp sourcePopulation 76 stderr 5.668 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/meta~1muse_spark_1_3` | | | | | | | | 22 | [GPT-5.5](https://benchize.com/models/gpt-5-5-xhigh.md)Extra high | 64.68% | SE ±5.53 pp | $6.05 | 925.37source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.681 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.052587 \| USD \| per_test \| Lower is better \| \| `latency` \| 925.371 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.53 pp sourcePopulation 76 stderr 5.533 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/openai~1gpt-5.5` | | | | | | | | 23 | [Claude Opus 4.7](https://benchize.com/models/claude-opus-4-7-max.md)Max | 64.52% | SE ±5.39 pp | $14.49 | 2,030.66source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.519 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 14.486164 \| USD \| per_test \| Lower is better \| \| `latency` \| 2030.658 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.39 pp sourcePopulation 76 stderr 5.387 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-opus-4-7` | | | | | | | | 24 | [GPT-5.6 Terra](https://benchize.com/models/gpt-5-6-terra-max.md)Max | 62.16% | SE ±8.4 pp | $3.97 | 884.57source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.157 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.968341 \| USD \| per_test \| Lower is better \| \| `latency` \| 884.572 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.4 pp sourcePopulation 76 stderr 8.404 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-terra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/openai~1gpt-5.6-terra` | | | | | | | | 25 | [GPT-6.1 Sol](https://benchize.com/models/gpt-6-1-sol-max.md)Max | 61.79% | SE ±8.55 pp | $2.74 | 1,969.17source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 61.788 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.739501 \| USD \| per_test \| Lower is better \| \| `latency` \| 1969.169 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.55 pp sourcePopulation 76 stderr 8.547 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6.1-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/openai~1gpt-6.1-sol` | | | | | | | | 26 | [Claude Sonnet 4.6](https://benchize.com/models/claude-sonnet-4-6-max.md)Max | 61.52% | SE ±5.97 pp | $8.99 | 3,683.56source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 61.52 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.994714 \| USD \| per_test \| Lower is better \| \| `latency` \| 3683.558 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning true Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.97 pp sourcePopulation 76 stderr 5.973 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-4-6 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/anthropic~1claude-sonnet-4-6` | | | | | | | | 27 | [DeepSeek V4 Flash 0731](https://benchize.com/models/deepseek-v4-flash-0731-high.md)High | 59.92% | SE ±6.9 pp | $0.36 | 1,361.08source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 59.918 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.355729 \| USD \| per_test \| Lower is better \| \| `latency` \| 1361.083 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.9 pp sourcePopulation 76 stderr 6.903 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-flash-0731 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/deepseek~1deepseek-v4-flash-0731` | | | | | | | | 28 | [DeepSeek V4 Pro](https://benchize.com/models/deepseek-v4-pro-max.md)Max | 57.43% | SE ±6.12 pp | $1.29 | 867.45source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.434 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.285259 \| USD \| per_test \| Lower is better \| \| `latency` \| 867.448 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.12 pp sourcePopulation 76 stderr 6.116 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/deepseek~1deepseek-v4-pro` | | | | | | | | 29 | [Muse Spark 1.2](https://benchize.com/models/muse-spark-1-2-xhigh.md)Extra high | 54.96% | SE ±8.78 pp | $2.87 | 1,114.73source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 54.957 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.865188 \| USD \| per_test \| Lower is better \| \| `latency` \| 1114.726 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.78 pp sourcePopulation 76 stderr 8.776 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_2 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/meta~1muse_spark_1_2` | | | | | | | | 30 | [GPT-5.6 Luna](https://benchize.com/models/gpt-5-6-luna-max.md)Max | 54.57% | SE ±8.83 pp | $0.73 | 621.72source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 54.573 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.729758 \| USD \| per_test \| Lower is better \| \| `latency` \| 621.715 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.83 pp sourcePopulation 76 stderr 8.831 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/openai~1gpt-5.6-luna` | | | | | | | | 31 | [MiniMax M3](https://benchize.com/models/minimax-m3.md) | 52.76% | SE ±7.58 pp | $1.44 | 1,421.6source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 52.763 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.442667 \| USD \| per_test \| Lower is better \| \| `latency` \| 1421.599 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 512000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.58 pp sourcePopulation 76 stderr 7.575 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/minimax~1MiniMax-M3` | | | | | | | | 32 | [Muse Spark 1.1](https://benchize.com/models/muse-spark-1-1-xhigh.md)Extra high | 52.72% | SE ±7.88 pp | $1.86 | 948.28source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 52.72 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.857162 \| USD \| per_test \| Lower is better \| \| `latency` \| 948.276 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 256000 provider Meta reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.88 pp sourcePopulation 76 stderr 7.879 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/meta~1muse_spark_1_1` | | | | | | | | 33 | [DeepSeek V4.1 Flash 1017](https://benchize.com/models/deepseek-v4-1-flash-1017-max.md)Max | 49.77% | SE ±8.07 pp | $0.34 | 815.93source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 49.774 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.343239 \| USD \| per_test \| Lower is better \| \| `latency` \| 815.93 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.07 pp sourcePopulation 76 stderr 8.069 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash-1017 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/deepseek~1deepseek-v4.1-flash-1017` | | | | | | | | 34 | [MiMo-V2.5-Pro](https://benchize.com/models/mimo-v2-5-pro.md) | 47.34% | SE ±7.04 pp | $0.4 | 1,597.32source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 47.341 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.399754 \| USD \| per_test \| Lower is better \| \| `latency` \| 1597.315 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.04 pp sourcePopulation 76 stderr 7.039 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/xiaomi~1mimo-v2.5-pro` | | | | | | | | 35 | [MiMo-V2.5](https://benchize.com/models/mimo-v2-5.md) | 47.17% | SE ±7.15 pp | $0.12 | 1,464.97source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 47.165 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.117056 \| USD \| per_test \| Lower is better \| \| `latency` \| 1464.972 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.15 pp sourcePopulation 76 stderr 7.148 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/xiaomi~1mimo-v2.5` | | | | | | | | 36 | [Gemini 3.5 Flash-Lite](https://benchize.com/models/gemini-3-5-flash-lite-high.md)High | 46.42% | SE ±7.78 pp | $1.78 | 591.25source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 46.417 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.779222 \| USD \| per_test \| Lower is better \| \| `latency` \| 591.245 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.78 pp sourcePopulation 76 stderr 7.777 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3.5-flash-lite` | | | | | | | | 37 | [Gemini 3.1 Pro Preview](https://benchize.com/models/gemini-3-1-pro-preview-high.md)High | 44.51% | SE ±7.62 pp | $7.47 | 768.42source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 44.505 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.466867 \| USD \| per_test \| Lower is better \| \| `latency` \| 768.417 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.62 pp sourcePopulation 76 stderr 7.615 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3.1-pro-preview` | | | | | | | | 38 | [DeepSeek V4.1 Flash](https://benchize.com/models/deepseek-v4-1-flash-high.md)High | 43.93% | SE ±9.09 pp | $0.22 | 630.8source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 43.925 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.218978 \| USD \| per_test \| Lower is better \| \| `latency` \| 630.8 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.09 pp sourcePopulation 76 stderr 9.09 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/deepseek~1deepseek-v4.1-flash` | | | | | | | | 39 | [DeepSeek V4 Pro 0813](https://benchize.com/models/deepseek-v4-pro-0813-max.md)Max | 42.15% | SE ±8.03 pp | $1.07 | 1,746.6source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 42.147 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.073792 \| USD \| per_test \| Lower is better \| \| `latency` \| 1746.604 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.03 pp sourcePopulation 76 stderr 8.028 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro-0813 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/deepseek~1deepseek-v4-pro-0813` | | | | | | | | 40 | [Muse Spark Open](https://benchize.com/models/muse-spark-open-xhigh.md)Extra high | 40.99% | SE ±8.4 pp | $6.33 | 4,497.27source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 40.991 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.33156 \| USD \| per_test \| Lower is better \| \| `latency` \| 4497.271 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Together AI reasoning Not reported Reasoning effort xhigh Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.4 pp sourcePopulation 76 stderr 8.397 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-spark-open Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/meta~1muse-spark-open` | | | | | | | | 41 | [Step 5 Preview](https://benchize.com/models/step-5-preview.md) | 40.63% | SE ±8.57 pp | $1.34 | 2,163.54source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 40.631 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.341743 \| USD \| per_test \| Lower is better \| \| `latency` \| 2163.542 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 1024000 provider Stepfun reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.57 pp sourcePopulation 76 stderr 8.571 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label stepfun/step-5-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/stepfun~1step-5-preview` | | | | | | | | 42 | [Nemotron 3 Ultra](https://benchize.com/models/nemotron-3-ultra-550b-a55b.md) | 32.45% | SE ±7.63 pp | — | 1,598.87source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 32.452 \| percent \| reported \| Higher is better \| \| `latency` \| 1598.874 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.63 pp sourcePopulation 76 stderr 7.631 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | 43 | [Gemini 3 Flash Preview](https://benchize.com/models/gemini-3-flash-preview-high.md)High | 31.3% | SE ±6.08 pp | $0.97 | 374.15source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 31.299 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.966148 \| USD \| per_test \| Lower is better \| \| `latency` \| 374.146 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.08 pp sourcePopulation 76 stderr 6.076 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3-flash-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3-flash-preview` | | | | | | | | 44 | [MiniMax M2.7](https://benchize.com/models/minimax-m2-7.md) | 23.58% | SE ±5.91 pp | $1.3 | 998.21source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 23.584 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.301526 \| USD \| per_test \| Lower is better \| \| `latency` \| 998.214 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.91 pp sourcePopulation 76 stderr 5.91 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M2.7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/minimax~1MiniMax-M2.7` | | | | | | | | 45 | [Mistral Medium 3.5](https://benchize.com/models/mistral-medium-3-5-high.md)High | 18.09% | SE ±5.44 pp | $10.64 | 3,251.3source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 18.088 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.642102 \| USD \| per_test \| Lower is better \| \| `latency` \| 3251.299 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider Mistral AI reasoning Not reported Reasoning effort high Temperature 0.7 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.44 pp sourcePopulation 76 stderr 5.441 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label mistralai/mistral-medium-3.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/mistralai~1mistral-medium-3.5` | | | | | | | | 46 | [Mercury 2.5](https://benchize.com/models/mercury-2-5-high.md)High | 11.52% | SE ±4.02 pp | $0.48 | 317.21source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 11.518 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.481209 \| USD \| per_test \| Lower is better \| \| `latency` \| 317.21 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Inception reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.02 pp sourcePopulation 76 stderr 4.015 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label inception/mercury-2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/inception~1mercury-2.5` | | | | | | | | 47 | [Gemini 3.1 Flash Lite Preview](https://benchize.com/models/gemini-3-1-flash-lite-preview-high.md)High | 10.48% | SE ±3.48 pp | $0.18 | 101.86source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 10.48 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.182456 \| USD \| per_test \| Lower is better \| \| `latency` \| 101.863 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.48 pp sourcePopulation 76 stderr 3.475 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-flash-lite-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3.1-flash-lite-preview` | | | | | | | --- Cite as: EMB / comps — Benchize, https://benchize.com/benchmarks/vals-ai%3Aemb%3Acomps (retrieved 2026-10-21)