# EMB / dataroom > Source: Benchize (https://benchize.com) — https://benchize.com/benchmarks/vals-ai%3Aemb%3Adataroom. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/benchmarks/vals-ai%3Aemb%3Adataroom. Evaluating agents on Excel-based financial modeling tasks Canonical URL: https://benchize.com/benchmarks/vals-ai%3Aemb%3Adataroom Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Evaluating agents on Excel-based financial modeling tasks Professional work · Models tested: 47 · Score publishers: Vals AI ## Model results | # | Model | Score | ± | Cost | Latency | | | --- | --- | --- | --- | --- | --- | --- | | 1 | [Claude Opus 5](https://benchize.com/models/claude-opus-5-max.md)Max | 87.68% | SE ±3.69 pp | $5.42 | 1,405.31source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 87.681 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.420298 \| USD \| per_test \| Lower is better \| \| `latency` \| 1405.306 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.69 pp sourcePopulation 76 stderr 3.687 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-opus-5` | | | | | | | | 2 | [Claude Fable 5.1](https://benchize.com/models/claude-fable-5-1-max.md)Max | 86.74% | SE ±3.65 pp | $10.87 | 1,370.44source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 86.736 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 10.865348 \| USD \| per_test \| Lower is better \| \| `latency` \| 1370.435 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.65 pp sourcePopulation 76 stderr 3.653 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5-1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-fable-5-1` | | | | | | | | 3 | [Gemini 3.7 Flash](https://benchize.com/models/gemini-3-7-flash-high.md)High | 86.48% | SE ±4.6 pp | $3.88 | 456.1source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 86.481 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.877832 \| USD \| per_test \| Lower is better \| \| `latency` \| 456.096 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.6 pp sourcePopulation 76 stderr 4.597 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.7-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/google~1gemini-3.7-flash` | | | | | | | | 4 | [Claude Sonnet 5.5](https://benchize.com/models/claude-sonnet-5-5-max.md)Max | 86.27% | SE ±3.89 pp | $7.86 | 2,042.34source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 86.27 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.856024 \| USD \| per_test \| Lower is better \| \| `latency` \| 2042.335 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.89 pp sourcePopulation 76 stderr 3.889 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-sonnet-5-5` | | | | | | | | 5 | [Gemini 3.8 Flash](https://benchize.com/models/gemini-3-8-flash-high.md)High | 86.22% | SE ±3.73 pp | $8.51 | 955.74source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 86.218 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.512634 \| USD \| per_test \| Lower is better \| \| `latency` \| 955.744 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.73 pp sourcePopulation 76 stderr 3.731 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.8-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/google~1gemini-3.8-flash` | | | | | | | | 6 | [Claude Opus 4.8](https://benchize.com/models/claude-opus-4-8-max.md)Max | 85.95% | SE ±3.72 pp | $8.99 | 2,182.51source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.948 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.994203 \| USD \| per_test \| Lower is better \| \| `latency` \| 2182.513 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.72 pp sourcePopulation 76 stderr 3.719 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-8 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-opus-4-8` | | | | | | | | 7 | [Claude Haiku 5.5](https://benchize.com/models/claude-haiku-5-5-max.md)Max | 85.48% | SE ±3.79 pp | $1.41 | 1,325.14source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.482 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.408937 \| USD \| per_test \| Lower is better \| \| `latency` \| 1325.144 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.79 pp sourcePopulation 76 stderr 3.787 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-haiku-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-haiku-5-5` | | | | | | | | 8 | [Claude Fable 5](https://benchize.com/models/claude-fable-5-max.md)Max | 85.36% | SE ±4.5 pp | $8.35 | 1,261.12source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.362 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.349512 \| USD \| per_test \| Lower is better \| \| `latency` \| 1261.119 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.5 pp sourcePopulation 76 stderr 4.502 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-fable-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-fable-5` | | | | | | | | 9 | [GPT-5.6 Sol](https://benchize.com/models/gpt-5-6-sol-max.md)Max | 85.24% | SE ±3.94 pp | $4.74 | 539.54source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.243 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.744315 \| USD \| per_test \| Lower is better \| \| `latency` \| 539.538 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.94 pp sourcePopulation 76 stderr 3.94 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/openai~1gpt-5.6-sol` | | | | | | | | 10 | [Gemini 4 Argon](https://benchize.com/models/gemini-4-argon-high.md)High | 85.11% | SE ±4.49 pp | $7.11 | 1,426.81source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.109 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.109521 \| USD \| per_test \| Lower is better \| \| `latency` \| 1426.808 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 262144 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.49 pp sourcePopulation 76 stderr 4.487 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-4-argon Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/google~1gemini-4-argon` | | | | | | | | 11 | [Claude Sonnet 5](https://benchize.com/models/claude-sonnet-5-max.md)Max | 84.91% | SE ±4.74 pp | $8.19 | 2,740.68source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.909 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.190469 \| USD \| per_test \| Lower is better \| \| `latency` \| 2740.679 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.74 pp sourcePopulation 76 stderr 4.74 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-sonnet-5` | | | | | | | | 12 | [Claude Opus 5.5](https://benchize.com/models/claude-opus-5-5-max.md)Max | 84.87% | SE ±4.79 pp | $7.83 | 1,493.03source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.873 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.825198 \| USD \| per_test \| Lower is better \| \| `latency` \| 1493.027 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.79 pp sourcePopulation 76 stderr 4.792 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-5-5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-opus-5-5` | | | | | | | | 13 | [GPT-6.1 Sol](https://benchize.com/models/gpt-6-1-sol-max.md)Max | 84.48% | SE ±3.89 pp | $0.78 | 757.82source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.477 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.775076 \| USD \| per_test \| Lower is better \| \| `latency` \| 757.817 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.89 pp sourcePopulation 76 stderr 3.887 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6.1-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/openai~1gpt-6.1-sol` | | | | | | | | 14 | [Muse Spark 1.3](https://benchize.com/models/muse-spark-1-3-xhigh.md)Extra high | 84.19% | SE ±4.7 pp | $2.61 | 1,056.83source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.191 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.610351 \| USD \| per_test \| Lower is better \| \| `latency` \| 1056.83 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.7 pp sourcePopulation 76 stderr 4.701 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/meta~1muse_spark_1_3` | | | | | | | | 15 | [GPT-6 Sol](https://benchize.com/models/gpt-6-sol-max.md)Max | 83.92% | SE ±4.29 pp | $1.63 | 647.3source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.92 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.626948 \| USD \| per_test \| Lower is better \| \| `latency` \| 647.302 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.29 pp sourcePopulation 76 stderr 4.286 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-sol Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/openai~1gpt-6-sol` | | | | | | | | 16 | [GPT-5.6 Luna](https://benchize.com/models/gpt-5-6-luna-max.md)Max | 83.01% | SE ±5.09 pp | $0.28 | 541.84source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.009 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.282142 \| USD \| per_test \| Lower is better \| \| `latency` \| 541.843 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.09 pp sourcePopulation 76 stderr 5.093 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/openai~1gpt-5.6-luna` | | | | | | | | 17 | [DeepSeek V4.1 Flash 1017](https://benchize.com/models/deepseek-v4-1-flash-1017-max.md)Max | 82.82% | SE ±6.05 pp | $0.3 | 932.19source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.824 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.295267 \| USD \| per_test \| Lower is better \| \| `latency` \| 932.192 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.05 pp sourcePopulation 76 stderr 6.053 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash-1017 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/deepseek~1deepseek-v4.1-flash-1017` | | | | | | | | 18 | [MiMo-V2.6-Flash](https://benchize.com/models/mimo-v2-6-flash.md) | 82.47% | SE ±5.2 pp | $0.08 | 1,519.19source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.468 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.083356 \| USD \| per_test \| Lower is better \| \| `latency` \| 1519.189 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.2 pp sourcePopulation 76 stderr 5.202 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/xiaomi~1mimo-v2.6-flash` | | | | | | | | 19 | [GPT-5.6 Terra](https://benchize.com/models/gpt-5-6-terra-max.md)Max | 82.16% | SE ±5.74 pp | $1.67 | 1,061.01source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.161 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.665623 \| USD \| per_test \| Lower is better \| \| `latency` \| 1061.013 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.74 pp sourcePopulation 76 stderr 5.744 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.6-terra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/openai~1gpt-5.6-terra` | | | | | | | | 20 | [GPT-5.5](https://benchize.com/models/gpt-5-5-xhigh.md)Extra high | 81.23% | SE ±4.69 pp | $2.42 | 818.42source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 81.227 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.424232 \| USD \| per_test \| Lower is better \| \| `latency` \| 818.421 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.69 pp sourcePopulation 76 stderr 4.694 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-5.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/openai~1gpt-5.5` | | | | | | | | 21 | [Gemini 3.5 Flash](https://benchize.com/models/gemini-3-5-flash-high.md)High | 80.92% | SE ±5.38 pp | $4.45 | 1,239.06source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.921 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.451663 \| USD \| per_test \| Lower is better \| \| `latency` \| 1239.062 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.38 pp sourcePopulation 76 stderr 5.377 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/google~1gemini-3.5-flash` | | | | | | | | 22 | [Claude Opus 4.7](https://benchize.com/models/claude-opus-4-7-max.md)Max | 80.91% | SE ±7.19 pp | $9.82 | 1,823.46source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.91 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 9.816464 \| USD \| per_test \| Lower is better \| \| `latency` \| 1823.46 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.19 pp sourcePopulation 76 stderr 7.193 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-opus-4-7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-opus-4-7` | | | | | | | | 23 | [GPT-6 Astra](https://benchize.com/models/gpt-6-astra-max.md)Max | 79.32% | SE ±7.18 pp | $4.72 | 619.46source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 79.319 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.722862 \| USD \| per_test \| Lower is better \| \| `latency` \| 619.459 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.18 pp sourcePopulation 76 stderr 7.177 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-astra Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/openai~1gpt-6-astra` | | | | | | | | 24 | [Muse Wonder](https://benchize.com/models/muse-wonder-max.md)Max | 79.23% | SE ±7.02 pp | $5.37 | 835.94source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 79.232 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.374192 \| USD \| per_test \| Lower is better \| \| `latency` \| 835.941 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.02 pp sourcePopulation 76 stderr 7.021 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-wonder Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/meta~1muse-wonder` | | | | | | | | 25 | [Gemini 3.1 Pro Preview](https://benchize.com/models/gemini-3-1-pro-preview-high.md)High | 79.17% | SE ±4.88 pp | $1.64 | 1,109.18source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 79.17 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.64075 \| USD \| per_test \| Lower is better \| \| `latency` \| 1109.179 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.88 pp sourcePopulation 76 stderr 4.883 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/google~1gemini-3.1-pro-preview` | | | | | | | | 26 | [Muse Spark 1.1](https://benchize.com/models/muse-spark-1-1-xhigh.md)Extra high | 78.62% | SE ±5.76 pp | $0.75 | 706.31source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 78.618 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.749924 \| USD \| per_test \| Lower is better \| \| `latency` \| 706.309 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 256000 provider Meta reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.76 pp sourcePopulation 76 stderr 5.76 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_1 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/meta~1muse_spark_1_1` | | | | | | | | 27 | [Claude Sonnet 4.6](https://benchize.com/models/claude-sonnet-4-6-max.md)Max | 78.49% | SE ±6.34 pp | $5.1 | 2,415.19source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 78.493 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.102641 \| USD \| per_test \| Lower is better \| \| `latency` \| 2415.188 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning true Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.34 pp sourcePopulation 76 stderr 6.337 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label anthropic/claude-sonnet-4-6 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/anthropic~1claude-sonnet-4-6` | | | | | | | | 28 | [MiMo-V2.5-Pro](https://benchize.com/models/mimo-v2-5-pro.md) | 77.97% | SE ±6.07 pp | $0.14 | 1,336.47source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.965 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.144435 \| USD \| per_test \| Lower is better \| \| `latency` \| 1336.466 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.07 pp sourcePopulation 76 stderr 6.068 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/xiaomi~1mimo-v2.5-pro` | | | | | | | | 29 | [MiMo-V2.6-Pro](https://benchize.com/models/mimo-v2-6-pro.md) | 76.7% | SE ±7.55 pp | $0.26 | 3,356.07source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 76.703 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.25859 \| USD \| per_test \| Lower is better \| \| `latency` \| 3356.071 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.55 pp sourcePopulation 76 stderr 7.551 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.6-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/xiaomi~1mimo-v2.6-pro` | | | | | | | | 30 | [Step 5 Preview](https://benchize.com/models/step-5-preview.md) | 75.69% | SE ±7.97 pp | $1.13 | 2,146.88source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.687 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.130429 \| USD \| per_test \| Lower is better \| \| `latency` \| 2146.883 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 1024000 provider Stepfun reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.97 pp sourcePopulation 76 stderr 7.969 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label stepfun/step-5-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/stepfun~1step-5-preview` | | | | | | | | 31 | [GPT-6 Luna](https://benchize.com/models/gpt-6-luna-max.md)Max | 75.54% | SE ±7.88 pp | $0.08 | 687.98source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.536 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.083815 \| USD \| per_test \| Lower is better \| \| `latency` \| 687.984 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.88 pp sourcePopulation 76 stderr 7.878 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label openai/gpt-6-luna Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/openai~1gpt-6-luna` | | | | | | | | 32 | [MiMo-V2.5](https://benchize.com/models/mimo-v2-5.md) | 74.96% | SE ±6.27 pp | $0.07 | 1,305.62source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 74.957 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.067242 \| USD \| per_test \| Lower is better \| \| `latency` \| 1305.618 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.27 pp sourcePopulation 76 stderr 6.273 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label xiaomi/mimo-v2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/xiaomi~1mimo-v2.5` | | | | | | | | 33 | [DeepSeek V4.1 Flash](https://benchize.com/models/deepseek-v4-1-flash-high.md)High | 73.86% | SE ±8.05 pp | $0.2 | 695.25source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.863 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.196161 \| USD \| per_test \| Lower is better \| \| `latency` \| 695.253 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.05 pp sourcePopulation 76 stderr 8.047 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/deepseek~1deepseek-v4.1-flash` | | | | | | | | 34 | [Gemini 3.6 Flash](https://benchize.com/models/gemini-3-6-flash-high.md)High | 72.97% | SE ±9.18 pp | $2.58 | 569.26source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.965 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.583986 \| USD \| per_test \| Lower is better \| \| `latency` \| 569.256 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.18 pp sourcePopulation 76 stderr 9.182 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.6-flash Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/google~1gemini-3.6-flash` | | | | | | | | 35 | [DeepSeek V4 Flash 0731](https://benchize.com/models/deepseek-v4-flash-0731-high.md)High | 72.8% | SE ±7.81 pp | $0.3 | 1,440.12source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.803 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.298467 \| USD \| per_test \| Lower is better \| \| `latency` \| 1440.119 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.81 pp sourcePopulation 76 stderr 7.812 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-flash-0731 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/deepseek~1deepseek-v4-flash-0731` | | | | | | | | 36 | [Muse Spark Open](https://benchize.com/models/muse-spark-open-xhigh.md)Extra high | 70.87% | SE ±7.81 pp | $6.15 | 5,754.88source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.871 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.145752 \| USD \| per_test \| Lower is better \| \| `latency` \| 5754.881 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Together AI reasoning Not reported Reasoning effort xhigh Temperature 1 topP 1 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.81 pp sourcePopulation 76 stderr 7.808 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse-spark-open Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/meta~1muse-spark-open` | | | | | | | | 37 | [Muse Spark 1.2](https://benchize.com/models/muse-spark-1-2-xhigh.md)Extra high | 68.59% | SE ±9.53 pp | $2.26 | 963.17source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.59 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.260701 \| USD \| per_test \| Lower is better \| \| `latency` \| 963.165 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.53 pp sourcePopulation 76 stderr 9.526 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label meta/muse_spark_1_2 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/meta~1muse_spark_1_2` | | | | | | | | 38 | [DeepSeek V4 Pro 0813](https://benchize.com/models/deepseek-v4-pro-0813-max.md)Max | 66.88% | SE ±9.25 pp | $0.96 | 1,827.69source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.878 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.962733 \| USD \| per_test \| Lower is better \| \| `latency` \| 1827.686 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.25 pp sourcePopulation 76 stderr 9.246 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro-0813 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/deepseek~1deepseek-v4-pro-0813` | | | | | | | | 39 | [DeepSeek V4 Pro](https://benchize.com/models/deepseek-v4-pro-max.md)Max | 62.57% | SE ±9.25 pp | $0.8 | 745.65source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.572 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.801873 \| USD \| per_test \| Lower is better \| \| `latency` \| 745.646 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.25 pp sourcePopulation 76 stderr 9.249 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label deepseek/deepseek-v4-pro Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/deepseek~1deepseek-v4-pro` | | | | | | | | 40 | [Gemini 3.5 Flash-Lite](https://benchize.com/models/gemini-3-5-flash-lite-high.md)High | 57.69% | SE ±7.58 pp | $1.1 | 475.52source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.687 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.097176 \| USD \| per_test \| Lower is better \| \| `latency` \| 475.52 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.58 pp sourcePopulation 76 stderr 7.582 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/google~1gemini-3.5-flash-lite` | | | | | | | | 41 | [MiniMax M3](https://benchize.com/models/minimax-m3.md) | 52.74% | SE ±10.63 pp | $2.08 | 1,882.98source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 52.743 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.081819 \| USD \| per_test \| Lower is better \| \| `latency` \| 1882.98 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 512000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±10.63 pp sourcePopulation 76 stderr 10.633 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M3 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/minimax~1MiniMax-M3` | | | | | | | | 42 | [Nemotron 3 Ultra](https://benchize.com/models/nemotron-3-ultra-550b-a55b.md) | 38.06% | SE ±9.38 pp | — | 1,261.47source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 38.062 \| percent \| reported \| Higher is better \| \| `latency` \| 1261.468 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.38 pp sourcePopulation 76 stderr 9.384 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | 43 | [MiniMax M2.7](https://benchize.com/models/minimax-m2-7.md) | 37.03% | SE ±7.8 pp | $1.2 | 916.51source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 37.032 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.197173 \| USD \| per_test \| Lower is better \| \| `latency` \| 916.512 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.8 pp sourcePopulation 76 stderr 7.797 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label minimax/MiniMax-M2.7 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/minimax~1MiniMax-M2.7` | | | | | | | | 44 | [Gemini 3 Flash Preview](https://benchize.com/models/gemini-3-flash-preview-high.md)High | 36.63% | SE ±10.76 pp | $0.44 | 459.04source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 36.634 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.438128 \| USD \| per_test \| Lower is better \| \| `latency` \| 459.043 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±10.76 pp sourcePopulation 76 stderr 10.763 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3-flash-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/google~1gemini-3-flash-preview` | | | | | | | | 45 | [Mercury 2.5](https://benchize.com/models/mercury-2-5-high.md)High | 15.51% | SE ±5.7 pp | $0.38 | 362.75source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 15.513 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.379957 \| USD \| per_test \| Lower is better \| \| `latency` \| 362.75 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Inception reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.7 pp sourcePopulation 76 stderr 5.699 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label inception/mercury-2.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/inception~1mercury-2.5` | | | | | | | | 46 | [Gemini 3.1 Flash Lite Preview](https://benchize.com/models/gemini-3-1-flash-lite-preview-high.md)High | 15.18% | SE ±5.15 pp | $0.05 | 84.27source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 15.178 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.050422 \| USD \| per_test \| Lower is better \| \| `latency` \| 84.269 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.15 pp sourcePopulation 76 stderr 5.151 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-flash-lite-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/google~1gemini-3.1-flash-lite-preview` | | | | | | | | 47 | [Mistral Medium 3.5](https://benchize.com/models/mistral-medium-3-5-high.md)High | 13.4% | SE ±6.62 pp | $8.16 | 2,169.62source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 13.403 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.157684 \| USD \| per_test \| Lower is better \| \| `latency` \| 2169.616 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider Mistral AI reasoning Not reported Reasoning effort high Temperature 0.7 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.62 pp sourcePopulation 76 stderr 6.618 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label mistralai/mistral-medium-3.5 Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/mistralai~1mistral-medium-3.5` | | | | | | | --- Cite as: EMB / dataroom — Benchize, https://benchize.com/benchmarks/vals-ai%3Aemb%3Adataroom (retrieved 2026-10-21)