# EMB / operating > Source: Benchize (https://benchize.com) — https://benchize.com/benchmarks/vals-ai%3Aemb%3Aoperating. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/benchmarks/vals-ai%3Aemb%3Aoperating. Evaluating agents on Excel-based financial modeling tasks Canonical URL: https://benchize.com/benchmarks/vals-ai%3Aemb%3Aoperating Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Evaluating agents on Excel-based financial modeling tasks Professional work · Models tested: 47 · Score publishers: Vals AI ## Model results | # | Model | Score | ± | Cost | Latency | | | --- | --- | --- | --- | --- | --- | --- | | 1 | [Claude Opus 5.5](https://benchize.com/models/claude-opus-5-5-max.md)Max | 77.86% | SE ±4.59 pp | $8.54 | 1,547.51source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.855 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.542039 \| USD \| per_test \| Lower is better \| \| `latency` \| 1547.508 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.59 pp sourcePopulation 76 stderr 4.588 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-opus-5-5` | | | | | | | | 2 | [GPT-6 Astra](https://benchize.com/models/gpt-6-astra-max.md)Max | 75.44% | SE ±4.24 pp | $3.62 | 590.97source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.439 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.61903 \| USD \| per_test \| Lower is better \| \| `latency` \| 590.973 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.24 pp sourcePopulation 76 stderr 4.244 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-astra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/openai~1gpt-6-astra` | | | | | | | | 3 | [Claude Fable 5.1](https://benchize.com/models/claude-fable-5-1-max.md)Max | 74.85% | SE ±4.51 pp | $13.42 | 1,431.69source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 74.849 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 13.417399 \| USD \| per_test \| Lower is better \| \| `latency` \| 1431.686 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.51 pp sourcePopulation 76 stderr 4.508 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5-1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-fable-5-1` | | | | | | | | 4 | [GPT-6.1 Sol](https://benchize.com/models/gpt-6-1-sol-max.md)Max | 73.92% | SE ±4.25 pp | $0.71 | 777.6source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.92 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.708017 \| USD \| per_test \| Lower is better \| \| `latency` \| 777.595 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.25 pp sourcePopulation 76 stderr 4.252 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6.1-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/openai~1gpt-6.1-sol` | | | | | | | | 5 | [Claude Opus 5](https://benchize.com/models/claude-opus-5-max.md)Max | 73.55% | SE ±6.4 pp | $5.6 | 1,435.27source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.545 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.596663 \| USD \| per_test \| Lower is better \| \| `latency` \| 1435.273 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.4 pp sourcePopulation 76 stderr 6.401 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-opus-5` | | | | | | | | 6 | [GPT-6 Sol](https://benchize.com/models/gpt-6-sol-max.md)Max | 72.92% | SE ±5.02 pp | $0.93 | 552.8source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.923 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.926683 \| USD \| per_test \| Lower is better \| \| `latency` \| 552.8 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.02 pp sourcePopulation 76 stderr 5.016 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/openai~1gpt-6-sol` | | | | | | | | 7 | [Claude Fable 5](https://benchize.com/models/claude-fable-5-max.md)Max | 72.83% | SE ±5.14 pp | $10.77 | 1,414.13source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.826 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.765076 \| USD \| per_test \| Lower is better \| \| `latency` \| 1414.129 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.14 pp sourcePopulation 76 stderr 5.142 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-fable-5` | | | | | | | | 8 | [GPT-5.6 Luna](https://benchize.com/models/gpt-5-6-luna-max.md)Max | 72.7% | SE ±5.17 pp | $0.22 | 578.94source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.697 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.21852 \| USD \| per_test \| Lower is better \| \| `latency` \| 578.944 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.17 pp sourcePopulation 76 stderr 5.168 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/openai~1gpt-5.6-luna` | | | | | | | | 9 | [Claude Sonnet 5.5](https://benchize.com/models/claude-sonnet-5-5-max.md)Max | 71.56% | SE ±7.07 pp | $7.81 | 2,013.79source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 71.555 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.810405 \| USD \| per_test \| Lower is better \| \| `latency` \| 2013.788 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.07 pp sourcePopulation 76 stderr 7.067 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-sonnet-5-5` | | | | | | | | 10 | [GPT-6 Luna](https://benchize.com/models/gpt-6-luna-max.md)Max | 71.09% | SE ±4.56 pp | $0.07 | 730.61source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 71.088 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.070005 \| USD \| per_test \| Lower is better \| \| `latency` \| 730.612 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.56 pp sourcePopulation 76 stderr 4.562 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/openai~1gpt-6-luna` | | | | | | | | 11 | [Gemini 4 Argon](https://benchize.com/models/gemini-4-argon-high.md)High | 69.43% | SE ±5.48 pp | $10.71 | 2,370.25source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.433 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.710486 \| USD \| per_test \| Lower is better \| \| `latency` \| 2370.247 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 262144 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.48 pp sourcePopulation 76 stderr 5.478 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-4-argon Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/google~1gemini-4-argon` | | | | | | | | 12 | [Gemini 3.8 Flash](https://benchize.com/models/gemini-3-8-flash-high.md)High | 69.06% | SE ±5.61 pp | $6.43 | 621.68source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.063 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.429094 \| USD \| per_test \| Lower is better \| \| `latency` \| 621.677 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.61 pp sourcePopulation 76 stderr 5.613 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.8-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/google~1gemini-3.8-flash` | | | | | | | | 13 | [GPT-5.6 Terra](https://benchize.com/models/gpt-5-6-terra-max.md)Max | 68.27% | SE ±4.1 pp | $1.51 | 683.42source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.269 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.509413 \| USD \| per_test \| Lower is better \| \| `latency` \| 683.423 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.1 pp sourcePopulation 76 stderr 4.095 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-terra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/openai~1gpt-5.6-terra` | | | | | | | | 14 | [GPT-5.6 Sol](https://benchize.com/models/gpt-5-6-sol-max.md)Max | 67.26% | SE ±6.51 pp | $4.31 | 448.59source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 67.257 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.311989 \| USD \| per_test \| Lower is better \| \| `latency` \| 448.591 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.51 pp sourcePopulation 76 stderr 6.512 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/openai~1gpt-5.6-sol` | | | | | | | | 15 | [Claude Opus 4.8](https://benchize.com/models/claude-opus-4-8-max.md)Max | 66.7% | SE ±6.3 pp | $10.39 | 2,376.91source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.699 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.392758 \| USD \| per_test \| Lower is better \| \| `latency` \| 2376.906 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.3 pp sourcePopulation 76 stderr 6.296 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-8 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-opus-4-8` | | | | | | | | 16 | [DeepSeek V4.1 Flash 1017](https://benchize.com/models/deepseek-v4-1-flash-1017-max.md)Max | 66.08% | SE ±6.29 pp | $0.32 | 909.37source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.084 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.323942 \| USD \| per_test \| Lower is better \| \| `latency` \| 909.365 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.29 pp sourcePopulation 76 stderr 6.293 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash-1017 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/deepseek~1deepseek-v4.1-flash-1017` | | | | | | | | 17 | [DeepSeek V4.1 Flash](https://benchize.com/models/deepseek-v4-1-flash-high.md)High | 66.05% | SE ±6.29 pp | $0.21 | 687.31source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.048 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.206267 \| USD \| per_test \| Lower is better \| \| `latency` \| 687.305 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.29 pp sourcePopulation 76 stderr 6.291 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/deepseek~1deepseek-v4.1-flash` | | | | | | | | 18 | [MiMo-V2.6-Pro](https://benchize.com/models/mimo-v2-6-pro.md) | 65.48% | SE ±6.53 pp | $0.29 | 4,297.24source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.484 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.291384 \| USD \| per_test \| Lower is better \| \| `latency` \| 4297.238 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.53 pp sourcePopulation 76 stderr 6.526 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/xiaomi~1mimo-v2.6-pro` | | | | | | | | 19 | [Claude Sonnet 5](https://benchize.com/models/claude-sonnet-5-max.md)Max | 65.46% | SE ±6.28 pp | $7.96 | 2,461.86source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.455 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.956599 \| USD \| per_test \| Lower is better \| \| `latency` \| 2461.863 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.28 pp sourcePopulation 76 stderr 6.282 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-sonnet-5` | | | | | | | | 20 | [Muse Wonder](https://benchize.com/models/muse-wonder-max.md)Max | 65.35% | SE ±6.63 pp | $4.82 | 917.12source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.354 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.824542 \| USD \| per_test \| Lower is better \| \| `latency` \| 917.117 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.63 pp sourcePopulation 76 stderr 6.634 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-wonder Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/meta~1muse-wonder` | | | | | | | | 21 | [GPT-5.5](https://benchize.com/models/gpt-5-5-xhigh.md)Extra high | 64.48% | SE ±6.33 pp | $2.2 | 798.92source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.478 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.199544 \| USD \| per_test \| Lower is better \| \| `latency` \| 798.917 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.33 pp sourcePopulation 76 stderr 6.329 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/openai~1gpt-5.5` | | | | | | | | 22 | [Muse Spark Open](https://benchize.com/models/muse-spark-open-xhigh.md)Extra high | 63.15% | SE ±6.53 pp | $4.2 | 4,388.69source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.149 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.195928 \| USD \| per_test \| Lower is better \| \| `latency` \| 4388.685 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Together AI reasoning Not reported Reasoning effort xhigh Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.53 pp sourcePopulation 76 stderr 6.528 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-spark-open Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/meta~1muse-spark-open` | | | | | | | | 23 | [Step 5 Preview](https://benchize.com/models/step-5-preview.md) | 63.12% | SE ±6.01 pp | $1.05 | 2,211.96source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.12 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.049678 \| USD \| per_test \| Lower is better \| \| `latency` \| 2211.962 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 1024000 provider Stepfun reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.01 pp sourcePopulation 76 stderr 6.007 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label stepfun/step-5-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/stepfun~1step-5-preview` | | | | | | | | 24 | [Gemini 3.6 Flash](https://benchize.com/models/gemini-3-6-flash-high.md)High | 62.97% | SE ±6.58 pp | $3.53 | 712.89source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.97 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.53297 \| USD \| per_test \| Lower is better \| \| `latency` \| 712.889 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.58 pp sourcePopulation 76 stderr 6.584 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/google~1gemini-3.6-flash` | | | | | | | | 25 | [MiMo-V2.6-Flash](https://benchize.com/models/mimo-v2-6-flash.md) | 62.53% | SE ±5.85 pp | $0.08 | 1,022.35source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.525 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.083495 \| USD \| per_test \| Lower is better \| \| `latency` \| 1022.354 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.85 pp sourcePopulation 76 stderr 5.851 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/xiaomi~1mimo-v2.6-flash` | | | | | | | | 26 | [Claude Opus 4.7](https://benchize.com/models/claude-opus-4-7-max.md)Max | 61.18% | SE ±7.14 pp | $10.94 | 2,115.38source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 61.176 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.941835 \| USD \| per_test \| Lower is better \| \| `latency` \| 2115.382 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.14 pp sourcePopulation 76 stderr 7.144 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-opus-4-7` | | | | | | | | 27 | [DeepSeek V4 Pro 0813](https://benchize.com/models/deepseek-v4-pro-0813-max.md)Max | 60.8% | SE ±7.25 pp | $0.85 | 1,896.48source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 60.803 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.845142 \| USD \| per_test \| Lower is better \| \| `latency` \| 1896.479 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.25 pp sourcePopulation 76 stderr 7.251 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro-0813 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/deepseek~1deepseek-v4-pro-0813` | | | | | | | | 28 | [Muse Spark 1.2](https://benchize.com/models/muse-spark-1-2-xhigh.md)Extra high | 59.42% | SE ±6.67 pp | $2.94 | 1,023.14source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 59.424 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.944004 \| USD \| per_test \| Lower is better \| \| `latency` \| 1023.138 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.67 pp sourcePopulation 76 stderr 6.665 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_2 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/meta~1muse_spark_1_2` | | | | | | | | 29 | [Gemini 3.5 Flash](https://benchize.com/models/gemini-3-5-flash-high.md)High | 58.66% | SE ±7.14 pp | $3.65 | 904.09source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 58.658 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.646821 \| USD \| per_test \| Lower is better \| \| `latency` \| 904.09 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.14 pp sourcePopulation 76 stderr 7.143 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/google~1gemini-3.5-flash` | | | | | | | | 30 | [Gemini 3.7 Flash](https://benchize.com/models/gemini-3-7-flash-high.md)High | 58.41% | SE ±6.9 pp | $3.59 | 462.57source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 58.406 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.585435 \| USD \| per_test \| Lower is better \| \| `latency` \| 462.574 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.9 pp sourcePopulation 76 stderr 6.897 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.7-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/google~1gemini-3.7-flash` | | | | | | | | 31 | [Gemini 3.1 Pro Preview](https://benchize.com/models/gemini-3-1-pro-preview-high.md)High | 58.09% | SE ±6.21 pp | $1.56 | 506.51source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 58.087 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.560798 \| USD \| per_test \| Lower is better \| \| `latency` \| 506.512 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.21 pp sourcePopulation 76 stderr 6.208 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/google~1gemini-3.1-pro-preview` | | | | | | | | 32 | [Claude Sonnet 4.6](https://benchize.com/models/claude-sonnet-4-6-max.md)Max | 58.06% | SE ±7.22 pp | $5.62 | 2,854.46source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 58.056 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.620452 \| USD \| per_test \| Lower is better \| \| `latency` \| 2854.463 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning true Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.22 pp sourcePopulation 76 stderr 7.22 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-4-6 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-sonnet-4-6` | | | | | | | | 33 | [Muse Spark 1.1](https://benchize.com/models/muse-spark-1-1-xhigh.md)Extra high | 57.56% | SE ±7.51 pp | $0.79 | 801.96source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.562 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.788153 \| USD \| per_test \| Lower is better \| \| `latency` \| 801.963 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 256000 provider Meta reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.51 pp sourcePopulation 76 stderr 7.506 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/meta~1muse_spark_1_1` | | | | | | | | 34 | [Claude Haiku 5.5](https://benchize.com/models/claude-haiku-5-5-max.md)Max | 56.41% | SE ±9.21 pp | $1.13 | 1,052.67source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 56.409 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.128358 \| USD \| per_test \| Lower is better \| \| `latency` \| 1052.672 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.21 pp sourcePopulation 76 stderr 9.206 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-haiku-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/anthropic~1claude-haiku-5-5` | | | | | | | | 35 | [DeepSeek V4 Flash 0731](https://benchize.com/models/deepseek-v4-flash-0731-high.md)High | 55.98% | SE ±7.06 pp | $0.32 | 1,316.34source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 55.976 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.32347 \| USD \| per_test \| Lower is better \| \| `latency` \| 1316.344 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.06 pp sourcePopulation 76 stderr 7.061 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-flash-0731 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/deepseek~1deepseek-v4-flash-0731` | | | | | | | | 36 | [MiMo-V2.5-Pro](https://benchize.com/models/mimo-v2-5-pro.md) | 55.18% | SE ±7.44 pp | $0.18 | 1,607.15source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 55.182 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.179455 \| USD \| per_test \| Lower is better \| \| `latency` \| 1607.152 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.44 pp sourcePopulation 76 stderr 7.443 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/xiaomi~1mimo-v2.5-pro` | | | | | | | | 37 | [Muse Spark 1.3](https://benchize.com/models/muse-spark-1-3-xhigh.md)Extra high | 53.81% | SE ±8.05 pp | $2.72 | 1,149.37source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 53.807 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.721422 \| USD \| per_test \| Lower is better \| \| `latency` \| 1149.374 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.05 pp sourcePopulation 76 stderr 8.05 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/meta~1muse_spark_1_3` | | | | | | | | 38 | [MiMo-V2.5](https://benchize.com/models/mimo-v2-5.md) | 51.54% | SE ±7.18 pp | $0.07 | 1,899.33source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 51.541 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.072876 \| USD \| per_test \| Lower is better \| \| `latency` \| 1899.325 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.18 pp sourcePopulation 76 stderr 7.178 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/xiaomi~1mimo-v2.5` | | | | | | | | 39 | [MiniMax M3](https://benchize.com/models/minimax-m3.md) | 48.86% | SE ±7.37 pp | $1.99 | 1,780.74source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 48.863 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.98786 \| USD \| per_test \| Lower is better \| \| `latency` \| 1780.736 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 512000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.37 pp sourcePopulation 76 stderr 7.366 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/minimax~1MiniMax-M3` | | | | | | | | 40 | [DeepSeek V4 Pro](https://benchize.com/models/deepseek-v4-pro-max.md)Max | 47.18% | SE ±6.65 pp | $0.54 | 685.55source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 47.176 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.540271 \| USD \| per_test \| Lower is better \| \| `latency` \| 685.554 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.65 pp sourcePopulation 76 stderr 6.647 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/deepseek~1deepseek-v4-pro` | | | | | | | | 41 | [Gemini 3.5 Flash-Lite](https://benchize.com/models/gemini-3-5-flash-lite-high.md)High | 32.74% | SE ±6.84 pp | $1.42 | 543.18source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 32.737 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.418567 \| USD \| per_test \| Lower is better \| \| `latency` \| 543.184 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.84 pp sourcePopulation 76 stderr 6.838 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/google~1gemini-3.5-flash-lite` | | | | | | | | 42 | [Nemotron 3 Ultra](https://benchize.com/models/nemotron-3-ultra-550b-a55b.md) | 28.12% | SE ±6.91 pp | — | 2,684.88source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 28.124 \| percent \| reported \| Higher is better \| \| `latency` \| 2684.882 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.91 pp sourcePopulation 76 stderr 6.907 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | 43 | [MiniMax M2.7](https://benchize.com/models/minimax-m2-7.md) | 26.68% | SE ±6.34 pp | $1.98 | 1,554.15source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 26.679 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.98355 \| USD \| per_test \| Lower is better \| \| `latency` \| 1554.153 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.34 pp sourcePopulation 76 stderr 6.337 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M2.7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/minimax~1MiniMax-M2.7` | | | | | | | | 44 | [Gemini 3 Flash Preview](https://benchize.com/models/gemini-3-flash-preview-high.md)High | 20.43% | SE ±7.16 pp | $0.37 | 295.09source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 20.428 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.367395 \| USD \| per_test \| Lower is better \| \| `latency` \| 295.092 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.16 pp sourcePopulation 76 stderr 7.156 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3-flash-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/google~1gemini-3-flash-preview` | | | | | | | | 45 | [Mistral Medium 3.5](https://benchize.com/models/mistral-medium-3-5-high.md)High | 10.18% | SE ±4.72 pp | $12.57 | 3,484.28source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 10.178 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.569555 \| USD \| per_test \| Lower is better \| \| `latency` \| 3484.28 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider Mistral AI reasoning Not reported Reasoning effort high Temperature 0.7 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.72 pp sourcePopulation 76 stderr 4.722 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label mistralai/mistral-medium-3.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/mistralai~1mistral-medium-3.5` | | | | | | | | 46 | [Gemini 3.1 Flash Lite Preview](https://benchize.com/models/gemini-3-1-flash-lite-preview-high.md)High | 5.4% | SE ±1.94 pp | $0.03 | 52.47source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 5.395 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.034484 \| USD \| per_test \| Lower is better \| \| `latency` \| 52.473 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.94 pp sourcePopulation 76 stderr 1.935 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-flash-lite-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/google~1gemini-3.1-flash-lite-preview` | | | | | | | | 47 | [Mercury 2.5](https://benchize.com/models/mercury-2-5-high.md)High | 2.63% | SE ±1.12 pp | $0.37 | 203.31source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 2.631 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.366292 \| USD \| per_test \| Lower is better \| \| `latency` \| 203.305 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Inception reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.12 pp sourcePopulation 76 stderr 1.122 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label inception/mercury-2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/inception~1mercury-2.5` | | | | | | | --- Cite as: EMB / operating — Benchize, https://benchize.com/benchmarks/vals-ai%3Aemb%3Aoperating (retrieved 2026-10-21)