# EMB / overall > Source: Benchize (https://benchize.com) — https://benchize.com/benchmarks/vals-ai%3Aemb%3Aoverall. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/benchmarks/vals-ai%3Aemb%3Aoverall. Evaluating agents on Excel-based financial modeling tasks Canonical URL: https://benchize.com/benchmarks/vals-ai%3Aemb%3Aoverall Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Evaluating agents on Excel-based financial modeling tasks Professional work · Models tested: 47 · Score publishers: Vals AI ## Model results | # | Model | Score | ± | Cost | Latency | | | --- | --- | --- | --- | --- | --- | --- | | 1 | [Claude Fable 5.1](https://benchize.com/models/claude-fable-5-1-max.md)Max | 76.67% | SE ±2.09 pp | $15.93 | 1,771.56source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 76.67 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 15.93249 \| USD \| per_test \| Lower is better \| \| `latency` \| 1771.558 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.09 pp sourcePopulation 76 stderr 2.085 tokenTotals {"cache_read_tokens":378586242,"cache_write_tokens":76164267,"input_tokens":454757375,"output_tokens":12499169,"reasoning_tokens":8150917} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5-1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-fable-5-1` | | | | | | | | 2 | [Claude Opus 5.5](https://benchize.com/models/claude-opus-5-5-max.md)Max | 75.94% | SE ±2.38 pp | $11.02 | 2,106.43source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.938 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 11.018974 \| USD \| per_test \| Lower is better \| \| `latency` \| 2106.434 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.38 pp sourcePopulation 76 stderr 2.383 tokenTotals {"cache_read_tokens":861165257,"cache_write_tokens":61067293,"input_tokens":922243024,"output_tokens":23704546,"reasoning_tokens":18569025} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-opus-5-5` | | | | | | | | 3 | [Claude Sonnet 5.5](https://benchize.com/models/claude-sonnet-5-5-max.md)Max | 75.71% | SE ±2.44 pp | $8.79 | 2,204.99source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.713 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.792863 \| USD \| per_test \| Lower is better \| \| `latency` \| 2204.991 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.44 pp sourcePopulation 76 stderr 2.437 tokenTotals {"cache_read_tokens":2311427347,"cache_write_tokens":56507215,"input_tokens":2367951142,"output_tokens":29328538,"reasoning_tokens":22320203} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-sonnet-5-5` | | | | | | | | 4 | [Gemini 4 Argon](https://benchize.com/models/gemini-4-argon-high.md)High | 75.24% | SE ±2.46 pp | $11.8 | 2,305.82source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.244 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 11.8029 \| USD \| per_test \| Lower is better \| \| `latency` \| 2305.821 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 262144 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.46 pp sourcePopulation 76 stderr 2.459 tokenTotals {"cache_read_tokens":945025185,"cache_write_tokens":null,"input_tokens":1127460088,"output_tokens":14257556,"reasoning_tokens":9497334} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-4-argon Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/google~1gemini-4-argon` | | | | | | | | 5 | [Claude Fable 5](https://benchize.com/models/claude-fable-5-max.md)Max | 73.67% | SE ±2.47 pp | $12.35 | 1,594.15source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.666 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.354419 \| USD \| per_test \| Lower is better \| \| `latency` \| 1594.153 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.47 pp sourcePopulation 76 stderr 2.468 tokenTotals {"cache_read_tokens":318538087,"cache_write_tokens":26523742,"input_tokens":345067688,"output_tokens":11953058,"reasoning_tokens":null} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-fable-5` | | | | | | | | 6 | [Claude Opus 5](https://benchize.com/models/claude-opus-5-max.md)Max | 73.56% | SE ±2.24 pp | $6.07 | 1,558.67source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.556 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.071566 \| USD \| per_test \| Lower is better \| \| `latency` \| 1558.673 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.24 pp sourcePopulation 76 stderr 2.24 tokenTotals {"cache_read_tokens":656949908,"cache_write_tokens":29058427,"input_tokens":686017503,"output_tokens":12323715,"reasoning_tokens":7957163} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-opus-5` | | | | | | | | 7 | [GPT-5.6 Sol](https://benchize.com/models/gpt-5-6-sol-max.md)Max | 72.34% | SE ±2.37 pp | $6.01 | 590.26source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.339 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.010773 \| USD \| per_test \| Lower is better \| \| `latency` \| 590.258 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.37 pp sourcePopulation 76 stderr 2.373 tokenTotals {"cache_read_tokens":276762112,"cache_write_tokens":null,"input_tokens":318866006,"output_tokens":5457571,"reasoning_tokens":3316634} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/openai~1gpt-5.6-sol` | | | | | | | | 8 | [Gemini 3.8 Flash](https://benchize.com/models/gemini-3-8-flash-high.md)High | 72.2% | SE ±2.42 pp | $8.24 | 769.2source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.196 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.243862 \| USD \| per_test \| Lower is better \| \| `latency` \| 769.197 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.42 pp sourcePopulation 76 stderr 2.42 tokenTotals {"cache_read_tokens":2787511478,"cache_write_tokens":null,"input_tokens":2960728904,"output_tokens":21722815,"reasoning_tokens":13162823} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.8-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/google~1gemini-3.8-flash` | | | | | | | | 9 | [GPT-6 Astra](https://benchize.com/models/gpt-6-astra-max.md)Max | 71.7% | SE ±2.48 pp | $5.83 | 804.28source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 71.7 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.827063 \| USD \| per_test \| Lower is better \| \| `latency` \| 804.28 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.48 pp sourcePopulation 76 stderr 2.475 tokenTotals {"cache_read_tokens":104449111,"cache_write_tokens":null,"input_tokens":123989946,"output_tokens":5542631,"reasoning_tokens":3051513} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-astra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/openai~1gpt-6-astra` | | | | | | | | 10 | [GPT-6 Sol](https://benchize.com/models/gpt-6-sol-max.md)Max | 71.53% | SE ±2.35 pp | $1.28 | 621.74source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 71.53 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.276882 \| USD \| per_test \| Lower is better \| \| `latency` \| 621.736 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.35 pp sourcePopulation 76 stderr 2.351 tokenTotals {"cache_read_tokens":204982294,"cache_write_tokens":11114719,"input_tokens":216104819,"output_tokens":5241605,"reasoning_tokens":3222968} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/openai~1gpt-6-sol` | | | | | | | | 11 | [Gemini 3.7 Flash](https://benchize.com/models/gemini-3-7-flash-high.md)High | 71.33% | SE ±2.26 pp | $5.69 | 650.91source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 71.326 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.694001 \| USD \| per_test \| Lower is better \| \| `latency` \| 650.914 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.26 pp sourcePopulation 76 stderr 2.262 tokenTotals {"cache_read_tokens":1546827567,"cache_write_tokens":null,"input_tokens":1686304951,"output_tokens":18606388,"reasoning_tokens":7782800} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.7-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/google~1gemini-3.7-flash` | | | | | | | | 12 | [GPT-6.1 Sol](https://benchize.com/models/gpt-6-1-sol-max.md)Max | 70.77% | SE ±2.71 pp | $1.14 | 1,107.81source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.773 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.143199 \| USD \| per_test \| Lower is better \| \| `latency` \| 1107.811 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.71 pp sourcePopulation 76 stderr 2.712 tokenTotals {"cache_read_tokens":112922114,"cache_write_tokens":16703171,"input_tokens":129631093,"output_tokens":6056928,"reasoning_tokens":3495567} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6.1-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/openai~1gpt-6.1-sol` | | | | | | | | 13 | [Claude Opus 4.8](https://benchize.com/models/claude-opus-4-8-max.md)Max | 69.37% | SE ±2.61 pp | $12.06 | 2,632.29source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.373 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.05844 \| USD \| per_test \| Lower is better \| \| `latency` \| 2632.285 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.61 pp sourcePopulation 76 stderr 2.606 tokenTotals {"cache_read_tokens":790543547,"cache_write_tokens":43067000,"input_tokens":833619980,"output_tokens":22136588,"reasoning_tokens":null} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-8 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-opus-4-8` | | | | | | | | 14 | [GPT-6 Luna](https://benchize.com/models/gpt-6-luna-max.md)Max | 68.52% | SE ±2.81 pp | $0.14 | 1,041.05source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.515 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.141708 \| USD \| per_test \| Lower is better \| \| `latency` \| 1041.052 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.81 pp sourcePopulation 76 stderr 2.809 tokenTotals {"cache_read_tokens":332678280,"cache_write_tokens":29598418,"input_tokens":362285754,"output_tokens":11007243,"reasoning_tokens":8590626} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/openai~1gpt-6-luna` | | | | | | | | 15 | [GPT-5.6 Luna](https://benchize.com/models/gpt-5-6-luna-max.md)Max | 67.12% | SE ±2.94 pp | $0.37 | 660.05source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 67.116 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.366502 \| USD \| per_test \| Lower is better \| \| `latency` \| 660.047 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.94 pp sourcePopulation 76 stderr 2.936 tokenTotals {"cache_read_tokens":604381577,"cache_write_tokens":null,"input_tokens":653086981,"output_tokens":6955414,"reasoning_tokens":4412805} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/openai~1gpt-5.6-luna` | | | | | | | | 16 | [Muse Wonder](https://benchize.com/models/muse-wonder-max.md)Max | 67.08% | SE ±2.89 pp | $5.6 | 1,034.44source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 67.084 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.599337 \| USD \| per_test \| Lower is better \| \| `latency` \| 1034.437 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.89 pp sourcePopulation 76 stderr 2.891 tokenTotals {"cache_read_tokens":1021915980,"cache_write_tokens":0,"input_tokens":1072958990,"output_tokens":8128857,"reasoning_tokens":5046443} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-wonder Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/meta~1muse-wonder` | | | | | | | | 17 | [Claude Sonnet 5](https://benchize.com/models/claude-sonnet-5-max.md)Max | 66.32% | SE ±3.01 pp | $10.29 | 2,985.7source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.316 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.293794 \| USD \| per_test \| Lower is better \| \| `latency` \| 2985.704 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.01 pp sourcePopulation 76 stderr 3.006 tokenTotals {"cache_read_tokens":3034388542,"cache_write_tokens":64983558,"input_tokens":3099396585,"output_tokens":25999381,"reasoning_tokens":null} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-sonnet-5` | | | | | | | | 18 | [GPT-5.6 Terra](https://benchize.com/models/gpt-5-6-terra-max.md)Max | 66.21% | SE ±3.11 pp | $2.19 | 813.02source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.205 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.193037 \| USD \| per_test \| Lower is better \| \| `latency` \| 813.024 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.11 pp sourcePopulation 76 stderr 3.11 tokenTotals {"cache_read_tokens":286789081,"cache_write_tokens":null,"input_tokens":314065634,"output_tokens":7024530,"reasoning_tokens":4854711} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-terra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/openai~1gpt-5.6-terra` | | | | | | | | 19 | [MiMo-V2.6-Flash](https://benchize.com/models/mimo-v2-6-flash.md) | 65.46% | SE ±2.71 pp | $0.11 | 1,521.91source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.459 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.108634 \| USD \| per_test \| Lower is better \| \| `latency` \| 1521.906 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.71 pp sourcePopulation 76 stderr 2.709 tokenTotals {"cache_read_tokens":861570496,"cache_write_tokens":null,"input_tokens":891374421,"output_tokens":16056047,"reasoning_tokens":12655956} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/xiaomi~1mimo-v2.6-flash` | | | | | | | | 20 | [Gemini 3.6 Flash](https://benchize.com/models/gemini-3-6-flash-high.md)High | 65.41% | SE ±2.65 pp | $3.89 | 726.48source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.411 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.893069 \| USD \| per_test \| Lower is better \| \| `latency` \| 726.48 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.65 pp sourcePopulation 76 stderr 2.646 tokenTotals {"cache_read_tokens":972645922,"cache_write_tokens":null,"input_tokens":1058720610,"output_tokens":15239728,"reasoning_tokens":6777930} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/google~1gemini-3.6-flash` | | | | | | | | 21 | [GPT-5.5](https://benchize.com/models/gpt-5-5-xhigh.md)Extra high | 64.54% | SE ±2.87 pp | $3.27 | 910.58source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.543 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.273352 \| USD \| per_test \| Lower is better \| \| `latency` \| 910.575 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.87 pp sourcePopulation 76 stderr 2.869 tokenTotals {"cache_read_tokens":167235072,"cache_write_tokens":null,"input_tokens":183023742,"output_tokens":5181007,"reasoning_tokens":3094376} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/openai~1gpt-5.5` | | | | | | | | 22 | [Claude Opus 4.7](https://benchize.com/models/claude-opus-4-7-max.md)Max | 63.73% | SE ±3.11 pp | $12.76 | 2,315.6source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.728 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.756352 \| USD \| per_test \| Lower is better \| \| `latency` \| 2315.595 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.11 pp sourcePopulation 76 stderr 3.106 tokenTotals {"cache_read_tokens":1174122388,"cache_write_tokens":40413529,"input_tokens":1214546122,"output_tokens":18458044,"reasoning_tokens":10154511} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-opus-4-7` | | | | | | | | 23 | [Gemini 3.5 Flash](https://benchize.com/models/gemini-3-5-flash-high.md)High | 63.55% | SE ±2.75 pp | $5.17 | 1,123.56source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.548 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.173115 \| USD \| per_test \| Lower is better \| \| `latency` \| 1123.555 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.75 pp sourcePopulation 76 stderr 2.752 tokenTotals {"cache_read_tokens":1520002205,"cache_write_tokens":null,"input_tokens":1617865425,"output_tokens":16984726,"reasoning_tokens":10956816} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/google~1gemini-3.5-flash` | | | | | | | | 24 | [MiMo-V2.6-Pro](https://benchize.com/models/mimo-v2-6-pro.md) | 62.86% | SE ±3.14 pp | $0.33 | 3,313.92source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.858 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.328999 \| USD \| per_test \| Lower is better \| \| `latency` \| 3313.924 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.14 pp sourcePopulation 76 stderr 3.141 tokenTotals {"cache_read_tokens":813521152,"cache_write_tokens":null,"input_tokens":847310790,"output_tokens":17554862,"reasoning_tokens":14594001} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/xiaomi~1mimo-v2.6-pro` | | | | | | | | 25 | [Muse Spark 1.3](https://benchize.com/models/muse-spark-1-3-xhigh.md)Extra high | 62.71% | SE ±2.96 pp | $3.55 | 1,375source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.712 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.552349 \| USD \| per_test \| Lower is better \| \| `latency` \| 1374.999 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.96 pp sourcePopulation 76 stderr 2.956 tokenTotals {"cache_read_tokens":null,"cache_write_tokens":null,"input_tokens":1152349029,"output_tokens":12397789,"reasoning_tokens":7178696} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/meta~1muse_spark_1_3` | | | | | | | | 26 | [DeepSeek V4.1 Flash 1017](https://benchize.com/models/deepseek-v4-1-flash-1017-max.md)Max | 61.81% | SE ±3.19 pp | $0.36 | 1,007.46source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 61.805 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.355142 \| USD \| per_test \| Lower is better \| \| `latency` \| 1007.461 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.19 pp sourcePopulation 76 stderr 3.192 tokenTotals {"cache_read_tokens":2076686592,"cache_write_tokens":null,"input_tokens":2089267477,"output_tokens":16658390,"reasoning_tokens":11880969} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash-1017 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/deepseek~1deepseek-v4.1-flash-1017` | | | | | | | | 27 | [Claude Sonnet 4.6](https://benchize.com/models/claude-sonnet-4-6-max.md)Max | 60.15% | SE ±3.28 pp | $7.23 | 3,255.85source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 60.154 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.22787 \| USD \| per_test \| Lower is better \| \| `latency` \| 3255.853 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning true Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.28 pp sourcePopulation 76 stderr 3.276 tokenTotals {"cache_read_tokens":854339701,"cache_write_tokens":43790957,"input_tokens":898138518,"output_tokens":21113414,"reasoning_tokens":null} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-4-6 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-sonnet-4-6` | | | | | | | | 28 | [Claude Haiku 5.5](https://benchize.com/models/claude-haiku-5-5-max.md)Max | 59.02% | SE ±4.17 pp | $1.15 | 1,082.9source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 59.015 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.146892 \| USD \| per_test \| Lower is better \| \| `latency` \| 1082.902 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.17 pp sourcePopulation 76 stderr 4.17 tokenTotals {"cache_read_tokens":1207542704,"cache_write_tokens":32752108,"input_tokens":1240305362,"output_tokens":21428591,"reasoning_tokens":16654436} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-haiku-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/anthropic~1claude-haiku-5-5` | | | | | | | | 29 | [DeepSeek V4.1 Flash](https://benchize.com/models/deepseek-v4-1-flash-high.md)High | 57.21% | SE ±3.23 pp | $0.23 | 780.46source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.208 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.234152 \| USD \| per_test \| Lower is better \| \| `latency` \| 780.458 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.23 pp sourcePopulation 76 stderr 3.227 tokenTotals {"cache_read_tokens":884618493,"cache_write_tokens":null,"input_tokens":893420650,"output_tokens":13279281,"reasoning_tokens":10184478} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/deepseek~1deepseek-v4.1-flash` | | | | | | | | 30 | [Step 5 Preview](https://benchize.com/models/step-5-preview.md) | 57.12% | SE ±3.17 pp | $1.33 | 2,615.1source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.116 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.329318 \| USD \| per_test \| Lower is better \| \| `latency` \| 2615.096 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 1024000 provider Stepfun reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.17 pp sourcePopulation 76 stderr 3.165 tokenTotals {"cache_read_tokens":1227785728,"cache_write_tokens":null,"input_tokens":1262287639,"output_tokens":14703425,"reasoning_tokens":0} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label stepfun/step-5-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/stepfun~1step-5-preview` | | | | | | | | 31 | [Muse Spark 1.2](https://benchize.com/models/muse-spark-1-2-xhigh.md)Extra high | 56.98% | SE ±3.13 pp | $2.31 | 1,027.48source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 56.978 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.313636 \| USD \| per_test \| Lower is better \| \| `latency` \| 1027.476 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.13 pp sourcePopulation 76 stderr 3.127 tokenTotals {"cache_read_tokens":769467634,"cache_write_tokens":null,"input_tokens":829404547,"output_tokens":10741088,"reasoning_tokens":5180580} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_2 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/meta~1muse_spark_1_2` | | | | | | | | 32 | [DeepSeek V4 Flash 0731](https://benchize.com/models/deepseek-v4-flash-0731-high.md)High | 56.98% | SE ±3.39 pp | $0.38 | 1,574.1source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 56.977 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.383709 \| USD \| per_test \| Lower is better \| \| `latency` \| 1574.097 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.39 pp sourcePopulation 76 stderr 3.391 tokenTotals {"cache_read_tokens":1008679680,"cache_write_tokens":null,"input_tokens":1018385162,"output_tokens":15135599,"reasoning_tokens":11153857} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-flash-0731 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/deepseek~1deepseek-v4-flash-0731` | | | | | | | | 33 | [Muse Spark 1.1](https://benchize.com/models/muse-spark-1-1-xhigh.md)Extra high | 56.36% | SE ±3.15 pp | $1.11 | 891.51source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 56.361 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.112498 \| USD \| per_test \| Lower is better \| \| `latency` \| 891.512 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 256000 provider Meta reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.15 pp sourcePopulation 76 stderr 3.153 tokenTotals {"cache_read_tokens":350123312,"cache_write_tokens":null,"input_tokens":368051977,"output_tokens":9069516,"reasoning_tokens":4627614} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/meta~1muse_spark_1_1` | | | | | | | | 34 | [MiMo-V2.5-Pro](https://benchize.com/models/mimo-v2-5-pro.md) | 55.23% | SE ±3.11 pp | $0.22 | 1,747.98source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 55.228 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.221264 \| USD \| per_test \| Lower is better \| \| `latency` \| 1747.98 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.11 pp sourcePopulation 76 stderr 3.112 tokenTotals {"cache_read_tokens":430971584,"cache_write_tokens":null,"input_tokens":455429765,"output_tokens":11928848,"reasoning_tokens":6975136} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/xiaomi~1mimo-v2.5-pro` | | | | | | | | 35 | [MiMo-V2.5](https://benchize.com/models/mimo-v2-5.md) | 55.09% | SE ±3.13 pp | $0.08 | 1,837.95source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 55.092 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.083304 \| USD \| per_test \| Lower is better \| \| `latency` \| 1837.949 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.13 pp sourcePopulation 76 stderr 3.127 tokenTotals {"cache_read_tokens":null,"cache_write_tokens":null,"input_tokens":599516539,"output_tokens":12515553,"reasoning_tokens":6917639} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/xiaomi~1mimo-v2.5` | | | | | | | | 36 | [Muse Spark Open](https://benchize.com/models/muse-spark-open-xhigh.md)Extra high | 53.67% | SE ±3.2 pp | $5.35 | 5,078.61source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 53.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.348821 \| USD \| per_test \| Lower is better \| \| `latency` \| 5078.614 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Together AI reasoning Not reported Reasoning effort xhigh Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.2 pp sourcePopulation 76 stderr 3.196 tokenTotals {"cache_read_tokens":642573184,"cache_write_tokens":null,"input_tokens":973042320,"output_tokens":8495845,"reasoning_tokens":4888586} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-spark-open Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/meta~1muse-spark-open` | | | | | | | | 37 | [DeepSeek V4 Pro 0813](https://benchize.com/models/deepseek-v4-pro-0813-max.md)Max | 52.8% | SE ±3.06 pp | $1.08 | 2,247.24source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 52.803 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.08141 \| USD \| per_test \| Lower is better \| \| `latency` \| 2247.236 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.06 pp sourcePopulation 76 stderr 3.059 tokenTotals {"cache_read_tokens":855589504,"cache_write_tokens":null,"input_tokens":864870937,"output_tokens":15254147,"reasoning_tokens":11469965} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro-0813 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/deepseek~1deepseek-v4-pro-0813` | | | | | | | | 38 | [Gemini 3.1 Pro Preview](https://benchize.com/models/gemini-3-1-pro-preview-high.md)High | 52.62% | SE ±2.97 pp | $3.77 | 708.58source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 52.616 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.769246 \| USD \| per_test \| Lower is better \| \| `latency` \| 708.582 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.97 pp sourcePopulation 76 stderr 2.974 tokenTotals {"cache_read_tokens":656983985,"cache_write_tokens":null,"input_tokens":703037870,"output_tokens":5442617,"reasoning_tokens":2522069} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | 39 | [DeepSeek V4 Pro](https://benchize.com/models/deepseek-v4-pro-max.md)Max | 51.62% | SE ±3.05 pp | $0.8 | 822.14source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 51.623 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.796757 \| USD \| per_test \| Lower is better \| \| `latency` \| 822.144 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.05 pp sourcePopulation 76 stderr 3.053 tokenTotals {"cache_read_tokens":527403648,"cache_write_tokens":null,"input_tokens":541981728,"output_tokens":9803133,"reasoning_tokens":3184172} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/deepseek~1deepseek-v4-pro` | | | | | | | | 40 | [MiniMax M3](https://benchize.com/models/minimax-m3.md) | 47.76% | SE ±2.95 pp | $2.09 | 1,890.34source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 47.759 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.091762 \| USD \| per_test \| Lower is better \| \| `latency` \| 1890.343 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 512000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.95 pp sourcePopulation 76 stderr 2.946 tokenTotals {"cache_read_tokens":1372552733,"cache_write_tokens":0,"input_tokens":1398807070,"output_tokens":13708644,"reasoning_tokens":null} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/minimax~1MiniMax-M3` | | | | | | | | 41 | [Gemini 3.5 Flash-Lite](https://benchize.com/models/gemini-3-5-flash-lite-high.md)High | 43.23% | SE ±3.11 pp | $1.23 | 493.63source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 43.231 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.231783 \| USD \| per_test \| Lower is better \| \| `latency` \| 493.632 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.11 pp sourcePopulation 76 stderr 3.105 tokenTotals {"cache_read_tokens":1299388807,"cache_write_tokens":null,"input_tokens":1498941214,"output_tokens":10717779,"reasoning_tokens":4101020} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | 42 | [Nemotron 3 Ultra](https://benchize.com/models/nemotron-3-ultra-550b-a55b.md) | 31.74% | SE ±2.81 pp | — | 2,382.36source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 31.735 \| percent \| reported \| Higher is better \| \| `latency` \| 2382.355 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.81 pp sourcePopulation 76 stderr 2.805 tokenTotals {"cache_read_tokens":153934848,"cache_write_tokens":null,"input_tokens":591493477,"output_tokens":12813242,"reasoning_tokens":null} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | 43 | [MiniMax M2.7](https://benchize.com/models/minimax-m2-7.md) | 28.45% | SE ±2.43 pp | $1.66 | 1,207.26source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 28.451 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.658686 \| USD \| per_test \| Lower is better \| \| `latency` \| 1207.259 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.43 pp sourcePopulation 76 stderr 2.431 tokenTotals {"cache_read_tokens":26409876,"cache_write_tokens":3330859,"input_tokens":547048102,"output_tokens":8578062,"reasoning_tokens":1143349} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M2.7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/minimax~1MiniMax-M2.7` | | | | | | | | 44 | [Gemini 3 Flash Preview](https://benchize.com/models/gemini-3-flash-preview-high.md)High | 26.02% | SE ±2.91 pp | $0.45 | 328.69source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 26.017 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.453183 \| USD \| per_test \| Lower is better \| \| `latency` \| 328.69 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.91 pp sourcePopulation 76 stderr 2.912 tokenTotals {"cache_read_tokens":210062842,"cache_write_tokens":null,"input_tokens":252451404,"output_tokens":4842419,"reasoning_tokens":1535428} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3-flash-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/google~1gemini-3-flash-preview` | | | | | | | | 45 | [Mistral Medium 3.5](https://benchize.com/models/mistral-medium-3-5-high.md)High | 11.19% | SE ±1.64 pp | $10.6 | 3,082.12source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 11.185 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.603072 \| USD \| per_test \| Lower is better \| \| `latency` \| 3082.124 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider Mistral AI reasoning Not reported Reasoning effort high Temperature 0.7 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.64 pp sourcePopulation 76 stderr 1.637 tokenTotals {"cache_read_tokens":null,"cache_write_tokens":null,"input_tokens":428119963,"output_tokens":6269299,"reasoning_tokens":null} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label mistralai/mistral-medium-3.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/mistralai~1mistral-medium-3.5` | | | | | | | | 46 | [Mercury 2.5](https://benchize.com/models/mercury-2-5-high.md)High | 8.79% | SE ±1.28 pp | $0.42 | 300.76source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 8.789 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.417608 \| USD \| per_test \| Lower is better \| \| `latency` \| 300.757 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Inception reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.28 pp sourcePopulation 76 stderr 1.275 tokenTotals {"cache_read_tokens":238888010,"cache_write_tokens":null,"input_tokens":407236592,"output_tokens":5531340,"reasoning_tokens":5399594} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label inception/mercury-2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/inception~1mercury-2.5` | | | | | | | | 47 | [Gemini 3.1 Flash Lite Preview](https://benchize.com/models/gemini-3-1-flash-lite-preview-high.md)High | 8.63% | SE ±1.09 pp | $0.07 | 67.01source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 8.629 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.069424 \| USD \| per_test \| Lower is better \| \| `latency` \| 67.007 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.09 pp sourcePopulation 76 stderr 1.088 tokenTotals {"cache_read_tokens":62784855,"cache_write_tokens":null,"input_tokens":77708032,"output_tokens":1187254,"reasoning_tokens":588837} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-flash-lite-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/overall/google~1gemini-3.1-flash-lite-preview` | | | | | | | --- Cite as: EMB / overall — Benchize, https://benchize.com/benchmarks/vals-ai%3Aemb%3Aoverall (retrieved 2026-10-21)