# EMB / scratch > Source: Benchize (https://benchize.com) — https://benchize.com/benchmarks/vals-ai%3Aemb%3Ascratch. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/benchmarks/vals-ai%3Aemb%3Ascratch. Evaluating agents on Excel-based financial modeling tasks Canonical URL: https://benchize.com/benchmarks/vals-ai%3Aemb%3Ascratch Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Evaluating agents on Excel-based financial modeling tasks Professional work · Models tested: 47 · Score publishers: Vals AI ## Model results | # | Model | Score | ± | Cost | Latency | | | --- | --- | --- | --- | --- | --- | --- | | 1 | [Claude Fable 5.1](https://benchize.com/models/claude-fable-5-1-max.md)Max | 72.59% | SE ±2.34 pp | $15.31 | 1,843.68source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.593 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 15.305899 \| USD \| per_test \| Lower is better \| \| `latency` \| 1843.681 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.34 pp sourcePopulation 76 stderr 2.339 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5-1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-fable-5-1` | | | | | | | | 2 | [Claude Opus 5](https://benchize.com/models/claude-opus-5-max.md)Max | 72.26% | SE ±2.57 pp | $5.64 | 1,622.5source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.255 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.637053 \| USD \| per_test \| Lower is better \| \| `latency` \| 1622.496 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.57 pp sourcePopulation 76 stderr 2.573 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-opus-5` | | | | | | | | 3 | [Gemini 4 Argon](https://benchize.com/models/gemini-4-argon-high.md)High | 70.8% | SE ±2.69 pp | $7.46 | 2,159.05source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.798 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.463166 \| USD \| per_test \| Lower is better \| \| `latency` \| 2159.046 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 262144 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.69 pp sourcePopulation 76 stderr 2.687 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-4-argon Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-4-argon` | | | | | | | | 4 | [Claude Sonnet 5.5](https://benchize.com/models/claude-sonnet-5-5-max.md)Max | 70.53% | SE ±3.15 pp | $8.58 | 2,251.75source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.53 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.575324 \| USD \| per_test \| Lower is better \| \| `latency` \| 2251.753 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.15 pp sourcePopulation 76 stderr 3.148 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-sonnet-5-5` | | | | | | | | 5 | [Claude Opus 5.5](https://benchize.com/models/claude-opus-5-5-max.md)Max | 70.51% | SE ±3.11 pp | $11.27 | 2,284.4source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.508 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 11.266649 \| USD \| per_test \| Lower is better \| \| `latency` \| 2284.395 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.11 pp sourcePopulation 76 stderr 3.11 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-opus-5-5` | | | | | | | | 6 | [Claude Fable 5](https://benchize.com/models/claude-fable-5-max.md)Max | 69.98% | SE ±2.99 pp | $12.07 | 1,639.29source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.978 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.066658 \| USD \| per_test \| Lower is better \| \| `latency` \| 1639.288 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.99 pp sourcePopulation 76 stderr 2.99 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-fable-5` | | | | | | | | 7 | [GPT-6 Sol](https://benchize.com/models/gpt-6-sol-max.md)Max | 69.48% | SE ±2.64 pp | $1.22 | 687.77source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.48 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.223773 \| USD \| per_test \| Lower is better \| \| `latency` \| 687.766 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.64 pp sourcePopulation 76 stderr 2.642 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/openai~1gpt-6-sol` | | | | | | | | 8 | [GPT-6 Astra](https://benchize.com/models/gpt-6-astra-max.md)Max | 69.09% | SE ±2.83 pp | $5.89 | 887.07source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.085 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.886751 \| USD \| per_test \| Lower is better \| \| `latency` \| 887.067 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.83 pp sourcePopulation 76 stderr 2.834 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-astra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/openai~1gpt-6-astra` | | | | | | | | 9 | [GPT-5.6 Sol](https://benchize.com/models/gpt-5-6-sol-max.md)Max | 69.08% | SE ±2.92 pp | $4.13 | 622.32source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.078 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.126546 \| USD \| per_test \| Lower is better \| \| `latency` \| 622.321 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.92 pp sourcePopulation 76 stderr 2.916 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/openai~1gpt-5.6-sol` | | | | | | | | 10 | [Muse Wonder](https://benchize.com/models/muse-wonder-max.md)Max | 69% | SE ±2.7 pp | $3.79 | 949.84source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.002 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.785539 \| USD \| per_test \| Lower is better \| \| `latency` \| 949.838 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.7 pp sourcePopulation 76 stderr 2.696 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-wonder Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/meta~1muse-wonder` | | | | | | | | 11 | [GPT-5.6 Luna](https://benchize.com/models/gpt-5-6-luna-max.md)Max | 68.79% | SE ±2.88 pp | $0.24 | 685.38source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.787 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.241679 \| USD \| per_test \| Lower is better \| \| `latency` \| 685.381 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.88 pp sourcePopulation 76 stderr 2.875 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/openai~1gpt-5.6-luna` | | | | | | | | 12 | [DeepSeek V4.1 Flash 1017](https://benchize.com/models/deepseek-v4-1-flash-1017-max.md)Max | 68.52% | SE ±2.53 pp | $0.35 | 1,011.15source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.523 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.347813 \| USD \| per_test \| Lower is better \| \| `latency` \| 1011.147 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.53 pp sourcePopulation 76 stderr 2.534 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash-1017 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/deepseek~1deepseek-v4.1-flash-1017` | | | | | | | | 13 | [Gemini 3.8 Flash](https://benchize.com/models/gemini-3-8-flash-high.md)High | 68.39% | SE ±2.73 pp | $6.98 | 724.8source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.39 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.984969 \| USD \| per_test \| Lower is better \| \| `latency` \| 724.799 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.73 pp sourcePopulation 76 stderr 2.731 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.8-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-3.8-flash` | | | | | | | | 14 | [Gemini 3.7 Flash](https://benchize.com/models/gemini-3-7-flash-high.md)High | 68.22% | SE ±2.56 pp | $5.32 | 724.52source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.222 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.318355 \| USD \| per_test \| Lower is better \| \| `latency` \| 724.523 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.56 pp sourcePopulation 76 stderr 2.559 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.7-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-3.7-flash` | | | | | | | | 15 | [Claude Opus 4.8](https://benchize.com/models/claude-opus-4-8-max.md)Max | 67.44% | SE ±2.85 pp | $11.14 | 2,538.57source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 67.437 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 11.138836 \| USD \| per_test \| Lower is better \| \| `latency` \| 2538.573 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.85 pp sourcePopulation 76 stderr 2.853 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-8 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-opus-4-8` | | | | | | | | 16 | [Claude Opus 4.7](https://benchize.com/models/claude-opus-4-7-max.md)Max | 67.05% | SE ±2.56 pp | $12.38 | 2,304.94source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 67.048 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.38453 \| USD \| per_test \| Lower is better \| \| `latency` \| 2304.936 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.56 pp sourcePopulation 76 stderr 2.557 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-opus-4-7` | | | | | | | | 17 | [GPT-5.5](https://benchize.com/models/gpt-5-5-xhigh.md)Extra high | 66.83% | SE ±2.47 pp | $3.41 | 947.62source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.833 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.412393 \| USD \| per_test \| Lower is better \| \| `latency` \| 947.624 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.47 pp sourcePopulation 76 stderr 2.465 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/openai~1gpt-5.5` | | | | | | | | 18 | [Step 5 Preview](https://benchize.com/models/step-5-preview.md) | 66.78% | SE ±2.48 pp | $1.22 | 2,506.87source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.777 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.217876 \| USD \| per_test \| Lower is better \| \| `latency` \| 2506.874 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 1024000 provider Stepfun reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.48 pp sourcePopulation 76 stderr 2.483 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label stepfun/step-5-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/stepfun~1step-5-preview` | | | | | | | | 19 | [GPT-6 Luna](https://benchize.com/models/gpt-6-luna-max.md)Max | 66.43% | SE ±2.84 pp | $0.11 | 996.44source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.431 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.108806 \| USD \| per_test \| Lower is better \| \| `latency` \| 996.439 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.84 pp sourcePopulation 76 stderr 2.843 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/openai~1gpt-6-luna` | | | | | | | | 20 | [GPT-6.1 Sol](https://benchize.com/models/gpt-6-1-sol-max.md)Max | 66.36% | SE ±3.36 pp | $1.29 | 1,255.71source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.358 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.290012 \| USD \| per_test \| Lower is better \| \| `latency` \| 1255.708 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.36 pp sourcePopulation 76 stderr 3.361 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6.1-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/openai~1gpt-6.1-sol` | | | | | | | | 21 | [MiMo-V2.6-Flash](https://benchize.com/models/mimo-v2-6-flash.md) | 66.15% | SE ±2.46 pp | $0.1 | 1,471.5source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.148 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.10021 \| USD \| per_test \| Lower is better \| \| `latency` \| 1471.499 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.46 pp sourcePopulation 76 stderr 2.462 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/xiaomi~1mimo-v2.6-flash` | | | | | | | | 22 | [DeepSeek V4 Pro 0813](https://benchize.com/models/deepseek-v4-pro-0813-max.md)Max | 66.11% | SE ±2.52 pp | $1.05 | 2,260.45source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.113 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.045429 \| USD \| per_test \| Lower is better \| \| `latency` \| 2260.453 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.52 pp sourcePopulation 76 stderr 2.524 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro-0813 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/deepseek~1deepseek-v4-pro-0813` | | | | | | | | 23 | [DeepSeek V4.1 Flash](https://benchize.com/models/deepseek-v4-1-flash-high.md)High | 66.08% | SE ±2.63 pp | $0.21 | 714.92source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.083 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.211272 \| USD \| per_test \| Lower is better \| \| `latency` \| 714.918 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.63 pp sourcePopulation 76 stderr 2.627 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/deepseek~1deepseek-v4.1-flash` | | | | | | | | 24 | [Muse Spark 1.3](https://benchize.com/models/muse-spark-1-3-xhigh.md)Extra high | 65.83% | SE ±2.7 pp | $2.55 | 1,298.37source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.829 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.548879 \| USD \| per_test \| Lower is better \| \| `latency` \| 1298.369 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.7 pp sourcePopulation 76 stderr 2.695 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/meta~1muse_spark_1_3` | | | | | | | | 25 | [Gemini 3.6 Flash](https://benchize.com/models/gemini-3-6-flash-high.md)High | 65.69% | SE ±2.48 pp | $3.55 | 753.53source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.691 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.552764 \| USD \| per_test \| Lower is better \| \| `latency` \| 753.531 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.48 pp sourcePopulation 76 stderr 2.475 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-3.6-flash` | | | | | | | | 26 | [Claude Sonnet 5](https://benchize.com/models/claude-sonnet-5-max.md)Max | 65.04% | SE ±3.21 pp | $10.48 | 3,133.34source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.037 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.476806 \| USD \| per_test \| Lower is better \| \| `latency` \| 3133.342 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.21 pp sourcePopulation 76 stderr 3.212 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-sonnet-5` | | | | | | | | 27 | [Claude Sonnet 4.6](https://benchize.com/models/claude-sonnet-4-6-max.md)Max | 64.75% | SE ±2.79 pp | $7.6 | 3,392.83source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.751 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.603845 \| USD \| per_test \| Lower is better \| \| `latency` \| 3392.83 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning true Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.79 pp sourcePopulation 76 stderr 2.791 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-4-6 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-sonnet-4-6` | | | | | | | | 28 | [Gemini 3.5 Flash](https://benchize.com/models/gemini-3-5-flash-high.md)High | 64% | SE ±2.42 pp | $4.74 | 1,075.4source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.744339 \| USD \| per_test \| Lower is better \| \| `latency` \| 1075.4 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.42 pp sourcePopulation 76 stderr 2.417 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-3.5-flash` | | | | | | | | 29 | [Muse Spark 1.2](https://benchize.com/models/muse-spark-1-2-xhigh.md)Extra high | 63.86% | SE ±2.88 pp | $1.75 | 970.68source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.861 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.749256 \| USD \| per_test \| Lower is better \| \| `latency` \| 970.676 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.88 pp sourcePopulation 76 stderr 2.882 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_2 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/meta~1muse_spark_1_2` | | | | | | | | 30 | [Muse Spark 1.1](https://benchize.com/models/muse-spark-1-1-xhigh.md)Extra high | 63.29% | SE ±2.57 pp | $1.02 | 868.09source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.285 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.015739 \| USD \| per_test \| Lower is better \| \| `latency` \| 868.085 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 256000 provider Meta reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.57 pp sourcePopulation 76 stderr 2.57 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/meta~1muse_spark_1_1` | | | | | | | | 31 | [GPT-5.6 Terra](https://benchize.com/models/gpt-5-6-terra-max.md)Max | 63.1% | SE ±3.02 pp | $2.13 | 953.39source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.096 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.127391 \| USD \| per_test \| Lower is better \| \| `latency` \| 953.388 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.02 pp sourcePopulation 76 stderr 3.022 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-terra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/openai~1gpt-5.6-terra` | | | | | | | | 32 | [Muse Spark Open](https://benchize.com/models/muse-spark-open-xhigh.md)Extra high | 62.65% | SE ±2.53 pp | $3.69 | 4,418.17source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.654 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.688473 \| USD \| per_test \| Lower is better \| \| `latency` \| 4418.173 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Together AI reasoning Not reported Reasoning effort xhigh Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.53 pp sourcePopulation 76 stderr 2.53 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-spark-open Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/meta~1muse-spark-open` | | | | | | | | 33 | [MiMo-V2.6-Pro](https://benchize.com/models/mimo-v2-6-pro.md) | 62.22% | SE ±3.51 pp | $0.29 | 3,235.5source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.215 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.287515 \| USD \| per_test \| Lower is better \| \| `latency` \| 3235.496 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.51 pp sourcePopulation 76 stderr 3.511 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/xiaomi~1mimo-v2.6-pro` | | | | | | | | 34 | [DeepSeek V4 Flash 0731](https://benchize.com/models/deepseek-v4-flash-0731-high.md)High | 61.43% | SE ±3.04 pp | $0.36 | 1,537.83source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 61.426 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.356 \| USD \| per_test \| Lower is better \| \| `latency` \| 1537.831 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.04 pp sourcePopulation 76 stderr 3.037 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-flash-0731 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/deepseek~1deepseek-v4-flash-0731` | | | | | | | | 35 | [MiniMax M3](https://benchize.com/models/minimax-m3.md) | 61.01% | SE ±2.56 pp | $1.85 | 1,976.51source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 61.012 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.854875 \| USD \| per_test \| Lower is better \| \| `latency` \| 1976.511 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 512000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.56 pp sourcePopulation 76 stderr 2.558 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/minimax~1MiniMax-M3` | | | | | | | | 36 | [MiMo-V2.5-Pro](https://benchize.com/models/mimo-v2-5-pro.md) | 57.92% | SE ±2.79 pp | $0.21 | 1,753.73source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.916 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.210941 \| USD \| per_test \| Lower is better \| \| `latency` \| 1753.73 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.79 pp sourcePopulation 76 stderr 2.79 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/xiaomi~1mimo-v2.5-pro` | | | | | | | | 37 | [Gemini 3.1 Pro Preview](https://benchize.com/models/gemini-3-1-pro-preview-high.md)High | 57.7% | SE ±2.83 pp | $3.15 | 624.19source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.704 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.148008 \| USD \| per_test \| Lower is better \| \| `latency` \| 624.193 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.83 pp sourcePopulation 76 stderr 2.832 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-3.1-pro-preview` | | | | | | | | 38 | [MiMo-V2.5](https://benchize.com/models/mimo-v2-5.md) | 57.15% | SE ±3.09 pp | $0.09 | 1,845.16source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.145 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.085787 \| USD \| per_test \| Lower is better \| \| `latency` \| 1845.163 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.09 pp sourcePopulation 76 stderr 3.085 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/xiaomi~1mimo-v2.5` | | | | | | | | 39 | [DeepSeek V4 Pro](https://benchize.com/models/deepseek-v4-pro-max.md)Max | 56.69% | SE ±2.82 pp | $0.79 | 866.49source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 56.692 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.794261 \| USD \| per_test \| Lower is better \| \| `latency` \| 866.492 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.82 pp sourcePopulation 76 stderr 2.818 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/deepseek~1deepseek-v4-pro` | | | | | | | | 40 | [Claude Haiku 5.5](https://benchize.com/models/claude-haiku-5-5-max.md)Max | 53.47% | SE ±4.97 pp | $1.18 | 1,134.68source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 53.47 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.175743 \| USD \| per_test \| Lower is better \| \| `latency` \| 1134.68 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.97 pp sourcePopulation 76 stderr 4.974 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-haiku-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/anthropic~1claude-haiku-5-5` | | | | | | | | 41 | [Gemini 3.5 Flash-Lite](https://benchize.com/models/gemini-3-5-flash-lite-high.md)High | 44.93% | SE ±2.94 pp | $1.15 | 477.72source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 44.928 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.149494 \| USD \| per_test \| Lower is better \| \| `latency` \| 477.721 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.94 pp sourcePopulation 76 stderr 2.935 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-3.5-flash-lite` | | | | | | | | 42 | [Nemotron 3 Ultra](https://benchize.com/models/nemotron-3-ultra-550b-a55b.md) | 44.31% | SE ±2.87 pp | — | 2,788.15source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 44.305 \| percent \| reported \| Higher is better \| \| `latency` \| 2788.147 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.87 pp sourcePopulation 76 stderr 2.869 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | 43 | [MiniMax M2.7](https://benchize.com/models/minimax-m2-7.md) | 36.72% | SE ±2.45 pp | $1.54 | 1,129.37source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 36.721 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.538796 \| USD \| per_test \| Lower is better \| \| `latency` \| 1129.369 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.45 pp sourcePopulation 76 stderr 2.448 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M2.7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/minimax~1MiniMax-M2.7` | | | | | | | | 44 | [Gemini 3 Flash Preview](https://benchize.com/models/gemini-3-flash-preview-high.md)High | 30.36% | SE ±3.66 pp | $0.4 | 302.05source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 30.356 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.402966 \| USD \| per_test \| Lower is better \| \| `latency` \| 302.045 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.66 pp sourcePopulation 76 stderr 3.656 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3-flash-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-3-flash-preview` | | | | | | | | 45 | [Mistral Medium 3.5](https://benchize.com/models/mistral-medium-3-5-high.md)High | 15.04% | SE ±2.8 pp | $10.9 | 3,219.68source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 15.038 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.895762 \| USD \| per_test \| Lower is better \| \| `latency` \| 3219.676 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider Mistral AI reasoning Not reported Reasoning effort high Temperature 0.7 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.8 pp sourcePopulation 76 stderr 2.802 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label mistralai/mistral-medium-3.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/mistralai~1mistral-medium-3.5` | | | | | | | | 46 | [Mercury 2.5](https://benchize.com/models/mercury-2-5-high.md)High | 12.44% | SE ±1.85 pp | $0.41 | 275source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 12.442 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.4053 \| USD \| per_test \| Lower is better \| \| `latency` \| 275.003 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Inception reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.85 pp sourcePopulation 76 stderr 1.846 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label inception/mercury-2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/inception~1mercury-2.5` | | | | | | | | 47 | [Gemini 3.1 Flash Lite Preview](https://benchize.com/models/gemini-3-1-flash-lite-preview-high.md)High | 9.07% | SE ±1.19 pp | $0.06 | 63.02source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 9.072 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.064241 \| USD \| per_test \| Lower is better \| \| `latency` \| 63.021 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.19 pp sourcePopulation 76 stderr 1.194 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-flash-lite-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-3.1-flash-lite-preview` | | | | | | | --- Cite as: EMB / scratch — Benchize, https://benchize.com/benchmarks/vals-ai%3Aemb%3Ascratch (retrieved 2026-10-21)