# EMB / ma > Source: Benchize (https://benchize.com) — https://benchize.com/benchmarks/vals-ai%3Aemb%3Ama. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/benchmarks/vals-ai%3Aemb%3Ama. Evaluating agents on Excel-based financial modeling tasks Canonical URL: https://benchize.com/benchmarks/vals-ai%3Aemb%3Ama Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Evaluating agents on Excel-based financial modeling tasks Professional work · Models tested: 47 · Score publishers: Vals AI ## Model results | # | Model | Score | ± | Cost | Latency | | | --- | --- | --- | --- | --- | --- | --- | | 1 | [GPT-6.1 Sol](https://benchize.com/models/gpt-6-1-sol-max.md)Max | 83.91% | SE ±2.94 pp | $0.9 | 939.37source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.906 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.897061 \| USD \| per_test \| Lower is better \| \| `latency` \| 939.372 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.94 pp sourcePopulation 76 stderr 2.941 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6.1-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/openai~1gpt-6.1-sol` | | | | | | | | 2 | [GPT-5.6 Sol](https://benchize.com/models/gpt-5-6-sol-max.md)Max | 83.87% | SE ±2.84 pp | $3.78 | 511.17source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.873 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.781208 \| USD \| per_test \| Lower is better \| \| `latency` \| 511.166 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.84 pp sourcePopulation 76 stderr 2.837 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/openai~1gpt-5.6-sol` | | | | | | | | 3 | [Gemini 3.7 Flash](https://benchize.com/models/gemini-3-7-flash-high.md)High | 83.36% | SE ±3.22 pp | $3.87 | 718.87source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.364 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.868043 \| USD \| per_test \| Lower is better \| \| `latency` \| 718.865 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.22 pp sourcePopulation 76 stderr 3.22 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.7-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3.7-flash` | | | | | | | | 4 | [GPT-6 Astra](https://benchize.com/models/gpt-6-astra-max.md)Max | 83.07% | SE ±2.89 pp | $5.24 | 718.57source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.066 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.243954 \| USD \| per_test \| Lower is better \| \| `latency` \| 718.565 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.89 pp sourcePopulation 76 stderr 2.888 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-astra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/openai~1gpt-6-astra` | | | | | | | | 5 | [GPT-6 Sol](https://benchize.com/models/gpt-6-sol-max.md)Max | 83.02% | SE ±2.86 pp | $1.17 | 677.53source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.021 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.171055 \| USD \| per_test \| Lower is better \| \| `latency` \| 677.534 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.86 pp sourcePopulation 76 stderr 2.861 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/openai~1gpt-6-sol` | | | | | | | | 6 | [Claude Sonnet 5.5](https://benchize.com/models/claude-sonnet-5-5-max.md)Max | 83% | SE ±3.32 pp | $7.9 | 2,061.35source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.002 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.900486 \| USD \| per_test \| Lower is better \| \| `latency` \| 2061.349 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.32 pp sourcePopulation 76 stderr 3.315 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-sonnet-5-5` | | | | | | | | 7 | [Claude Opus 5.5](https://benchize.com/models/claude-opus-5-5-max.md)Max | 82.31% | SE ±3.28 pp | $9.72 | 1,803.19source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.31 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 9.722535 \| USD \| per_test \| Lower is better \| \| `latency` \| 1803.192 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.28 pp sourcePopulation 76 stderr 3.275 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-opus-5-5` | | | | | | | | 8 | [Claude Fable 5.1](https://benchize.com/models/claude-fable-5-1-max.md)Max | 81.2% | SE ±3.27 pp | $14.03 | 1,368.61source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 81.202 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 14.03427 \| USD \| per_test \| Lower is better \| \| `latency` \| 1368.605 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.27 pp sourcePopulation 76 stderr 3.265 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5-1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-fable-5-1` | | | | | | | | 9 | [Gemini 3.8 Flash](https://benchize.com/models/gemini-3-8-flash-high.md)High | 81.03% | SE ±4.92 pp | $5.78 | 609.15source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 81.03 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.784968 \| USD \| per_test \| Lower is better \| \| `latency` \| 609.151 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.92 pp sourcePopulation 76 stderr 4.921 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.8-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3.8-flash` | | | | | | | | 10 | [Gemini 4 Argon](https://benchize.com/models/gemini-4-argon-high.md)High | 80.97% | SE ±4.89 pp | $9.01 | 2,261.07source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.97 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 9.012355 \| USD \| per_test \| Lower is better \| \| `latency` \| 2261.071 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 262144 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.89 pp sourcePopulation 76 stderr 4.891 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-4-argon Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-4-argon` | | | | | | | | 11 | [Claude Fable 5](https://benchize.com/models/claude-fable-5-max.md)Max | 80.57% | SE ±3.45 pp | $10.85 | 1,534.54source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.566 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.847761 \| USD \| per_test \| Lower is better \| \| `latency` \| 1534.541 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.45 pp sourcePopulation 76 stderr 3.449 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-fable-5` | | | | | | | | 12 | [Claude Opus 5](https://benchize.com/models/claude-opus-5-max.md)Max | 80.4% | SE ±3.13 pp | $4.46 | 1,346.24source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.399 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.455983 \| USD \| per_test \| Lower is better \| \| `latency` \| 1346.235 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.13 pp sourcePopulation 76 stderr 3.132 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-opus-5` | | | | | | | | 13 | [Claude Opus 4.8](https://benchize.com/models/claude-opus-4-8-max.md)Max | 80.09% | SE ±3.26 pp | $9.45 | 2,301.31source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.091 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 9.449638 \| USD \| per_test \| Lower is better \| \| `latency` \| 2301.312 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.26 pp sourcePopulation 76 stderr 3.258 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-8 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-opus-4-8` | | | | | | | | 14 | [GPT-5.6 Terra](https://benchize.com/models/gpt-5-6-terra-max.md)Max | 77.56% | SE ±4.86 pp | $1.29 | 762.85source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.558 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.28773 \| USD \| per_test \| Lower is better \| \| `latency` \| 762.847 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.86 pp sourcePopulation 76 stderr 4.857 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-terra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/openai~1gpt-5.6-terra` | | | | | | | | 15 | [Muse Wonder](https://benchize.com/models/muse-wonder-max.md)Max | 76.85% | SE ±6.61 pp | $4.18 | 857.04source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 76.852 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.182315 \| USD \| per_test \| Lower is better \| \| `latency` \| 857.035 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.61 pp sourcePopulation 76 stderr 6.608 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-wonder Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/meta~1muse-wonder` | | | | | | | | 16 | [Claude Sonnet 5](https://benchize.com/models/claude-sonnet-5-max.md)Max | 75.83% | SE ±4.7 pp | $8.34 | 2,458.64source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.832 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.337588 \| USD \| per_test \| Lower is better \| \| `latency` \| 2458.641 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.7 pp sourcePopulation 76 stderr 4.698 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-sonnet-5` | | | | | | | | 17 | [Gemini 3.6 Flash](https://benchize.com/models/gemini-3-6-flash-high.md)High | 75.65% | SE ±3.9 pp | $3.58 | 668.59source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.646 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.583839 \| USD \| per_test \| Lower is better \| \| `latency` \| 668.593 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.9 pp sourcePopulation 76 stderr 3.903 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3.6-flash` | | | | | | | | 18 | [MiMo-V2.6-Pro](https://benchize.com/models/mimo-v2-6-pro.md) | 74.43% | SE ±6.76 pp | $0.27 | 2,491.54source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 74.431 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.272898 \| USD \| per_test \| Lower is better \| \| `latency` \| 2491.543 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.76 pp sourcePopulation 76 stderr 6.756 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/xiaomi~1mimo-v2.6-pro` | | | | | | | | 19 | [GPT-6 Luna](https://benchize.com/models/gpt-6-luna-max.md)Max | 72.05% | SE ±6.69 pp | $0.09 | 754.2source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.051 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.085662 \| USD \| per_test \| Lower is better \| \| `latency` \| 754.204 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.69 pp sourcePopulation 76 stderr 6.691 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/openai~1gpt-6-luna` | | | | | | | | 20 | [Claude Opus 4.7](https://benchize.com/models/claude-opus-4-7-max.md)Max | 71.67% | SE ±5.33 pp | $9.2 | 2,053.28source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 71.668 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 9.204118 \| USD \| per_test \| Lower is better \| \| `latency` \| 2053.282 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.33 pp sourcePopulation 76 stderr 5.328 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-opus-4-7` | | | | | | | | 21 | [Claude Sonnet 4.6](https://benchize.com/models/claude-sonnet-4-6-max.md)Max | 70.5% | SE ±6.53 pp | $5.38 | 2,932.98source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.503 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.383689 \| USD \| per_test \| Lower is better \| \| `latency` \| 2932.982 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning true Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.53 pp sourcePopulation 76 stderr 6.534 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-4-6 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-sonnet-4-6` | | | | | | | | 22 | [GPT-5.6 Luna](https://benchize.com/models/gpt-5-6-luna-max.md)Max | 69.61% | SE ±7.28 pp | $0.2 | 524.15source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.61 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.19832 \| USD \| per_test \| Lower is better \| \| `latency` \| 524.146 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.28 pp sourcePopulation 76 stderr 7.279 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/openai~1gpt-5.6-luna` | | | | | | | | 23 | [Gemini 3.5 Flash](https://benchize.com/models/gemini-3-5-flash-high.md)High | 68.45% | SE ±6.09 pp | $4.12 | 1,014.82source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.445 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.118564 \| USD \| per_test \| Lower is better \| \| `latency` \| 1014.816 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.09 pp sourcePopulation 76 stderr 6.087 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3.5-flash` | | | | | | | | 24 | [GPT-5.5](https://benchize.com/models/gpt-5-5-xhigh.md)Extra high | 66.81% | SE ±8.09 pp | $2.6 | 866.36source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.813 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.601886 \| USD \| per_test \| Lower is better \| \| `latency` \| 866.362 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.09 pp sourcePopulation 76 stderr 8.09 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/openai~1gpt-5.5` | | | | | | | | 25 | [Step 5 Preview](https://benchize.com/models/step-5-preview.md) | 64.61% | SE ±8.54 pp | $1.11 | 2,412.2source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.61 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.114729 \| USD \| per_test \| Lower is better \| \| `latency` \| 2412.197 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 1024000 provider Stepfun reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.54 pp sourcePopulation 76 stderr 8.543 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label stepfun/step-5-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/stepfun~1step-5-preview` | | | | | | | | 26 | [Muse Spark 1.3](https://benchize.com/models/muse-spark-1-3-xhigh.md)Extra high | 64.37% | SE ±8.07 pp | $2.16 | 1,015.13source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.366 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.158016 \| USD \| per_test \| Lower is better \| \| `latency` \| 1015.132 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.07 pp sourcePopulation 76 stderr 8.072 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/meta~1muse_spark_1_3` | | | | | | | | 27 | [MiMo-V2.6-Flash](https://benchize.com/models/mimo-v2-6-flash.md) | 64.1% | SE ±8.18 pp | $0.09 | 1,173.96source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.098 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.089712 \| USD \| per_test \| Lower is better \| \| `latency` \| 1173.958 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.18 pp sourcePopulation 76 stderr 8.179 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/xiaomi~1mimo-v2.6-flash` | | | | | | | | 27 | [MiMo-V2.5-Pro](https://benchize.com/models/mimo-v2-5-pro.md) | 64.1% | SE ±7.42 pp | $0.17 | 1,662.4source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.098 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.166174 \| USD \| per_test \| Lower is better \| \| `latency` \| 1662.395 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.42 pp sourcePopulation 76 stderr 7.421 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/xiaomi~1mimo-v2.5-pro` | | | | | | | | 29 | [Muse Spark 1.2](https://benchize.com/models/muse-spark-1-2-xhigh.md)Extra high | 63.85% | SE ±8.63 pp | $1.48 | 848.88source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.851 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.481971 \| USD \| per_test \| Lower is better \| \| `latency` \| 848.879 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.63 pp sourcePopulation 76 stderr 8.634 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_2 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/meta~1muse_spark_1_2` | | | | | | | | 30 | [DeepSeek V4.1 Flash 1017](https://benchize.com/models/deepseek-v4-1-flash-1017-max.md)Max | 62.29% | SE ±9.83 pp | $0.31 | 914.97source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.29 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.307102 \| USD \| per_test \| Lower is better \| \| `latency` \| 914.971 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.83 pp sourcePopulation 76 stderr 9.825 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash-1017 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/deepseek~1deepseek-v4.1-flash-1017` | | | | | | | | 31 | [Muse Spark 1.1](https://benchize.com/models/muse-spark-1-1-xhigh.md)Extra high | 61.5% | SE ±9.31 pp | $0.69 | 808.49source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 61.504 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.691326 \| USD \| per_test \| Lower is better \| \| `latency` \| 808.487 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 256000 provider Meta reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.31 pp sourcePopulation 76 stderr 9.309 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/meta~1muse_spark_1_1` | | | | | | | | 32 | [MiMo-V2.5](https://benchize.com/models/mimo-v2-5.md) | 60.35% | SE ±7.5 pp | $0.07 | 1,845.11source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 60.354 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.06733 \| USD \| per_test \| Lower is better \| \| `latency` \| 1845.107 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.5 pp sourcePopulation 76 stderr 7.502 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/xiaomi~1mimo-v2.5` | | | | | | | | 33 | [Gemini 3.1 Pro Preview](https://benchize.com/models/gemini-3-1-pro-preview-high.md)High | 58.83% | SE ±7.83 pp | $6.07 | 646.47source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 58.833 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.070452 \| USD \| per_test \| Lower is better \| \| `latency` \| 646.47 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.83 pp sourcePopulation 76 stderr 7.829 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3.1-pro-preview` | | | | | | | | 34 | [DeepSeek V4.1 Flash](https://benchize.com/models/deepseek-v4-1-flash-high.md)High | 57.78% | SE ±9.4 pp | $0.19 | 661.5source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.776 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.194981 \| USD \| per_test \| Lower is better \| \| `latency` \| 661.499 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.4 pp sourcePopulation 76 stderr 9.403 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/deepseek~1deepseek-v4.1-flash` | | | | | | | | 35 | [Claude Haiku 5.5](https://benchize.com/models/claude-haiku-5-5-max.md)Max | 57.76% | SE ±10.5 pp | $0.98 | 1,011.49source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.764 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.98052 \| USD \| per_test \| Lower is better \| \| `latency` \| 1011.493 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±10.5 pp sourcePopulation 76 stderr 10.499 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-haiku-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/anthropic~1claude-haiku-5-5` | | | | | | | | 36 | [DeepSeek V4 Flash 0731](https://benchize.com/models/deepseek-v4-flash-0731-high.md)High | 54.76% | SE ±9.21 pp | $0.28 | 1,184.38source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 54.761 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.283682 \| USD \| per_test \| Lower is better \| \| `latency` \| 1184.383 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.21 pp sourcePopulation 76 stderr 9.208 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-flash-0731 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/deepseek~1deepseek-v4-flash-0731` | | | | | | | | 37 | [Muse Spark Open](https://benchize.com/models/muse-spark-open-xhigh.md)Extra high | 53.7% | SE ±9.12 pp | $3.86 | 5,014.64source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 53.698 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.855837 \| USD \| per_test \| Lower is better \| \| `latency` \| 5014.642 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Together AI reasoning Not reported Reasoning effort xhigh Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.12 pp sourcePopulation 76 stderr 9.124 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-spark-open Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/meta~1muse-spark-open` | | | | | | | | 38 | [DeepSeek V4 Pro 0813](https://benchize.com/models/deepseek-v4-pro-0813-max.md)Max | 53.06% | SE ±10.56 pp | $0.75 | 1,799.44source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 53.058 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.750715 \| USD \| per_test \| Lower is better \| \| `latency` \| 1799.44 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±10.56 pp sourcePopulation 76 stderr 10.557 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro-0813 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/deepseek~1deepseek-v4-pro-0813` | | | | | | | | 39 | [MiniMax M3](https://benchize.com/models/minimax-m3.md) | 48.37% | SE ±9.64 pp | $1.46 | 1,834.69source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 48.366 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.463202 \| USD \| per_test \| Lower is better \| \| `latency` \| 1834.694 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 512000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.64 pp sourcePopulation 76 stderr 9.642 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/minimax~1MiniMax-M3` | | | | | | | | 40 | [DeepSeek V4 Pro](https://benchize.com/models/deepseek-v4-pro-max.md)Max | 48.16% | SE ±9.25 pp | $0.65 | 764.84source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 48.156 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.645157 \| USD \| per_test \| Lower is better \| \| `latency` \| 764.843 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.25 pp sourcePopulation 76 stderr 9.249 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/deepseek~1deepseek-v4-pro` | | | | | | | | 41 | [Gemini 3.5 Flash-Lite](https://benchize.com/models/gemini-3-5-flash-lite-high.md)High | 43.21% | SE ±8 pp | $0.95 | 417.57source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 43.213 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.952387 \| USD \| per_test \| Lower is better \| \| `latency` \| 417.573 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8 pp sourcePopulation 76 stderr 7.997 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3.5-flash-lite` | | | | | | | | 42 | [Nemotron 3 Ultra](https://benchize.com/models/nemotron-3-ultra-550b-a55b.md) | 37.06% | SE ±7.57 pp | — | 2,678.05source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 37.061 \| percent \| reported \| Higher is better \| \| `latency` \| 2678.053 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.57 pp sourcePopulation 76 stderr 7.567 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | 43 | [MiniMax M2.7](https://benchize.com/models/minimax-m2-7.md) | 35.26% | SE ±6.11 pp | $1.35 | 1,046.46source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 35.256 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.350619 \| USD \| per_test \| Lower is better \| \| `latency` \| 1046.457 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.11 pp sourcePopulation 76 stderr 6.11 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M2.7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/minimax~1MiniMax-M2.7` | | | | | | | | 44 | [Gemini 3 Flash Preview](https://benchize.com/models/gemini-3-flash-preview-high.md)High | 34.41% | SE ±8.11 pp | $0.3 | 321.45source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 34.407 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.304123 \| USD \| per_test \| Lower is better \| \| `latency` \| 321.449 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.11 pp sourcePopulation 76 stderr 8.107 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3-flash-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3-flash-preview` | | | | | | | | 45 | [Mistral Medium 3.5](https://benchize.com/models/mistral-medium-3-5-high.md)High | 10.69% | SE ±4.89 pp | $8.97 | 2,712.8source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 10.694 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.967043 \| USD \| per_test \| Lower is better \| \| `latency` \| 2712.804 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider Mistral AI reasoning Not reported Reasoning effort high Temperature 0.7 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.89 pp sourcePopulation 76 stderr 4.89 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label mistralai/mistral-medium-3.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/mistralai~1mistral-medium-3.5` | | | | | | | | 46 | [Gemini 3.1 Flash Lite Preview](https://benchize.com/models/gemini-3-1-flash-lite-preview-high.md)High | 6.75% | SE ±1.42 pp | $0.03 | 55.63source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 6.751 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.034741 \| USD \| per_test \| Lower is better \| \| `latency` \| 55.632 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.42 pp sourcePopulation 76 stderr 1.416 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-flash-lite-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3.1-flash-lite-preview` | | | | | | | | 47 | [Mercury 2.5](https://benchize.com/models/mercury-2-5-high.md)High | 6.35% | SE ±2.25 pp | $0.57 | 395.7source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 6.35 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.572506 \| USD \| per_test \| Lower is better \| \| `latency` \| 395.701 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Inception reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.25 pp sourcePopulation 76 stderr 2.25 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label inception/mercury-2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/inception~1mercury-2.5` | | | | | | | --- Cite as: EMB / ma — Benchize, https://benchize.com/benchmarks/vals-ai%3Aemb%3Ama (retrieved 2026-10-21)