# Terminal-Bench 2.1 / easy > Source: Benchize (https://benchize.com) — https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Aeasy. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Aeasy. State-of-the-art set of difficult terminal-based tasks Canonical URL: https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Aeasy Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Conquered State-of-the-art set of difficult terminal-based tasks Agents & tools · Models tested: 43 · Score publishers: Vals AI ## Model results | # | Model | Score | ± | Cost | Latency | | | --- | --- | --- | --- | --- | --- | --- | | 1 | [Claude Sonnet 4.6](https://benchize.com/models/claude-sonnet-4-6-off.md)Non-thinking | 100% | SE ±0 pp | $12.74 | 15,669source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.735719 \| USD \| per_test \| Lower is better \| \| `latency` \| 15669.003 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning false Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label anthropic/claude-sonnet-4-6 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/anthropic~1claude-sonnet-4-6` | | | | | | | | 1 | [Claude Fable 5](https://benchize.com/models/claude-fable-5-max.md)Max | 100% | SE ±0 pp | $9.54 | 11,225.95source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 9.538742 \| USD \| per_test \| Lower is better \| \| `latency` \| 11225.953 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label anthropic/claude-fable-5 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/anthropic~1claude-fable-5` | | | | | | | | 1 | [GPT-5.6 Terra](https://benchize.com/models/gpt-5-6-terra-max.md)Max | 100% | SE ±0 pp | $0.25 | 11,564.19source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.248207 \| USD \| per_test \| Lower is better \| \| `latency` \| 11564.187 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label openai/gpt-5.6-terra Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/openai~1gpt-5.6-terra` | | | | | | | | 1 | [GPT-6 Sol](https://benchize.com/models/gpt-6-sol-max.md)Max | 100% | SE ±0 pp | $8.3 | 6,650source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 8.299794 \| USD \| per_test \| Lower is better \| \| `latency` \| 6649.999 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label openai/gpt-6-sol Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/openai~1gpt-6-sol` | | | | | | | | 1 | [Claude Fable 5.1](https://benchize.com/models/claude-fable-5-1-high.md)High | 100% | SE ±0 pp | $1.74 | 11,132.9source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.741133 \| USD \| per_test \| Lower is better \| \| `latency` \| 11132.898 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort high harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label anthropic/claude-fable-5-1 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/anthropic~1claude-fable-5-1` | | | | | | | | 1 | [Gemini 3.6 Flash](https://benchize.com/models/gemini-3-6-flash-high.md)High | 100% | SE ±0 pp | $26.13 | 12,285.42source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 26.130479 \| USD \| per_test \| Lower is better \| \| `latency` \| 12285.415 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3.6-flash Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3.6-flash` | | | | | | | | 1 | [MiMo-V2.5](https://benchize.com/models/mimo-v2-5.md) | 100% | SE ±0 pp | $0.48 | 17,249.06source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.481881 \| USD \| per_test \| Lower is better \| \| `latency` \| 17249.061 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label xiaomi/mimo-v2.5 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/xiaomi~1mimo-v2.5` | | | | | | | | 1 | [Gemini 3.5 Flash](https://benchize.com/models/gemini-3-5-flash-high.md)High | 100% | SE ±0 pp | $19.32 | 8,304.21source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 19.316978 \| USD \| per_test \| Lower is better \| \| `latency` \| 8304.209 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3.5-flash Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3.5-flash` | | | | | | | | 1 | [GPT-6 Astra](https://benchize.com/models/gpt-6-astra-max.md)Max | 100% | SE ±0 pp | $29.8 | 6,106.29source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 29.803716 \| USD \| per_test \| Lower is better \| \| `latency` \| 6106.294 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label openai/gpt-6-astra Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/openai~1gpt-6-astra` | | | | | | | | 1 | [Claude Opus 4.8](https://benchize.com/models/claude-opus-4-8-max.md)Max | 100% | SE ±0 pp | $32.17 | 20,690.32source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 32.170449 \| USD \| per_test \| Lower is better \| \| `latency` \| 20690.317 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label anthropic/claude-opus-4-8 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/anthropic~1claude-opus-4-8` | | | | | | | | 1 | [Gemini 3.8 Flash](https://benchize.com/models/gemini-3-8-flash-high.md)High | 100% | SE ±0 pp | $0.93 | 12,344.24source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.931088 \| USD \| per_test \| Lower is better \| \| `latency` \| 12344.24 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3.8-flash Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3.8-flash` | | | | | | | | 1 | [Claude Opus 5](https://benchize.com/models/claude-opus-5-high.md)High | 100% | SE ±0 pp | $0.45 | 16,752.73source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.44962 \| USD \| per_test \| Lower is better \| \| `latency` \| 16752.727 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort high harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label anthropic/claude-opus-5 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/anthropic~1claude-opus-5` | | | | | | | | 1 | [Claude Opus 5.5](https://benchize.com/models/claude-opus-5-5-high.md)High | 100% | SE ±0 pp | $0.3 | 8,523.51source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.30277 \| USD \| per_test \| Lower is better \| \| `latency` \| 8523.514 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort high harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label anthropic/claude-opus-5-5 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/anthropic~1claude-opus-5-5` | | | | | | | | 1 | [Claude Sonnet 5](https://benchize.com/models/claude-sonnet-5-max.md)Max | 100% | SE ±0 pp | $0.27 | 14,129.71source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 100 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.269427 \| USD \| per_test \| Lower is better \| \| `latency` \| 14129.714 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort max harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort max Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label anthropic/claude-sonnet-5 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/anthropic~1claude-sonnet-5` | | | | | | | | 15 | [Gemini 3.1 Pro Preview](https://benchize.com/models/gemini-3-1-pro-preview-high.md)High | 91.67% | SE ±8.33 pp | $12.96 | 9,128.05source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.964574 \| USD \| per_test \| Lower is better \| \| `latency` \| 9128.048 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3.1-pro-preview` | | | | | | | | 15 | [Gemini 3.7 Flash](https://benchize.com/models/gemini-3-7-flash-high.md)High | 91.67% | SE ±8.33 pp | $22.74 | 10,951.5source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 22.744373 \| USD \| per_test \| Lower is better \| \| `latency` \| 10951.504 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3.7-flash Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3.7-flash` | | | | | | | | 15 | [GPT-6 Luna](https://benchize.com/models/gpt-6-luna-max.md)Max | 91.67% | SE ±8.33 pp | $0.65 | 9,458.34source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.650194 \| USD \| per_test \| Lower is better \| \| `latency` \| 9458.34 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label openai/gpt-6-luna Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/openai~1gpt-6-luna` | | | | | | | | 15 | [Nemotron 3 Ultra](https://benchize.com/models/nemotron-3-ultra-550b-a55b.md) | 91.67% | SE ±8.33 pp | — | 12,996.57source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `latency` \| 12996.574 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | 15 | [Muse Spark 1.2](https://benchize.com/models/muse-spark-1-2-xhigh.md)Extra high | 91.67% | SE ±8.33 pp | $11.11 | 19,563.83source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 11.113242 \| USD \| per_test \| Lower is better \| \| `latency` \| 19563.827 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label meta/muse_spark_1_2 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/meta~1muse_spark_1_2` | | | | | | | | 15 | [GPT-5.6 Sol](https://benchize.com/models/gpt-5-6-sol-max.md)Max | 91.67% | SE ±8.33 pp | $22.77 | 8,295.21source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 22.768842 \| USD \| per_test \| Lower is better \| \| `latency` \| 8295.214 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label openai/gpt-5.6-sol Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/openai~1gpt-5.6-sol` | | | | | | | | 15 | [DeepSeek V4.1 Flash](https://benchize.com/models/deepseek-v4-1-flash-high.md)High | 91.67% | SE ±8.33 pp | $2.2 | 13,450.26source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.200009 \| USD \| per_test \| Lower is better \| \| `latency` \| 13450.264 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label deepseek/deepseek-v4.1-flash Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/deepseek~1deepseek-v4.1-flash` | | | | | | | | 15 | [Claude Sonnet 5.5](https://benchize.com/models/claude-sonnet-5-5-high.md)High | 91.67% | SE ±8.33 pp | $0.49 | 10,364.66source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.49001 \| USD \| per_test \| Lower is better \| \| `latency` \| 10364.662 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort high harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label anthropic/claude-sonnet-5-5 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/anthropic~1claude-sonnet-5-5` | | | | | | | | 15 | [DeepSeek V4 Flash 0731](https://benchize.com/models/deepseek-v4-flash-0731-off.md)Non-thinking | 91.67% | SE ±8.33 pp | $0.47 | 15,197.41source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.46877 \| USD \| per_test \| Lower is better \| \| `latency` \| 15197.407 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider DeepSeek reasoning false Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label deepseek/deepseek-v4-flash-0731 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/deepseek~1deepseek-v4-flash-0731` | | | | | | | | 15 | [Claude Opus 4.7](https://benchize.com/models/claude-opus-4-7-high.md)High | 91.67% | SE ±8.33 pp | $43.99 | 18,169.07source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 43.990177 \| USD \| per_test \| Lower is better \| \| `latency` \| 18169.067 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort high harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label anthropic/claude-opus-4-7 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/anthropic~1claude-opus-4-7` | | | | | | | | 25 | [MiMo-V2.6-Flash](https://benchize.com/models/mimo-v2-6-flash.md) | 83.33% | SE ±8.33 pp | $0.52 | 18,016.56source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.333 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.518875 \| USD \| per_test \| Lower is better \| \| `latency` \| 18016.562 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label xiaomi/mimo-v2.6-flash Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/xiaomi~1mimo-v2.6-flash` | | | | | | | | 25 | [MiniMax M3](https://benchize.com/models/minimax-m3.md) | 83.33% | SE ±8.33 pp | $4.58 | 22,771source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.333 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.582903 \| USD \| per_test \| Lower is better \| \| `latency` \| 22771.001 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 512000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label minimax/MiniMax-M3 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/minimax~1MiniMax-M3` | | | | | | | | 25 | [GPT-5.5](https://benchize.com/models/gpt-5-5-high.md)High | 83.33% | SE ±8.33 pp | $16.51 | 9,509.92source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.333 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 16.506894 \| USD \| per_test \| Lower is better \| \| `latency` \| 9509.922 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label openai/gpt-5.5 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/openai~1gpt-5.5` | | | | | | | | 25 | [GPT-5.6 Luna](https://benchize.com/models/gpt-5-6-luna-max.md)Max | 83.33% | SE ±16.67 pp | $0.02 | 7,915.47source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.333 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.020477 \| USD \| per_test \| Lower is better \| \| `latency` \| 7915.469 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±16.67 pp sourcePopulation 76 stderr 16.667 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label openai/gpt-5.6-luna Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/openai~1gpt-5.6-luna` | | | | | | | | 29 | [Gemini 3 Flash Preview](https://benchize.com/models/gemini-3-flash-preview-high.md)High | 75% | SE ±14.43 pp | $3.43 | 8,369.51source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.433448 \| USD \| per_test \| Lower is better \| \| `latency` \| 8369.51 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±14.43 pp sourcePopulation 76 stderr 14.434 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3-flash-preview Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3-flash-preview` | | | | | | | | 29 | [MiMo-V2.5-Pro](https://benchize.com/models/mimo-v2-5-pro.md) | 75% | SE ±25 pp | $0.97 | 19,274.13source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.970383 \| USD \| per_test \| Lower is better \| \| `latency` \| 19274.134 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±25 pp sourcePopulation 76 stderr 25 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label xiaomi/mimo-v2.5-pro Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/xiaomi~1mimo-v2.5-pro` | | | | | | | | 29 | [Laguna M 1](https://benchize.com/models/laguna-m-1.md) | 75% | SE ±0 pp | $6.69 | 26,913.78source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.687796 \| USD \| per_test \| Lower is better \| \| `latency` \| 26913.783 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 32768 provider Poolside reasoning Not reported Reasoning effort unknown Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label poolside/laguna-m.1 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/poolside~1laguna-m.1` | | | | | | | | 29 | [MiniMax M2.7](https://benchize.com/models/minimax-m2-7.md) | 75% | SE ±0 pp | $1.47 | 18,081.2source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.468763 \| USD \| per_test \| Lower is better \| \| `latency` \| 18081.196 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 196608 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label minimax/MiniMax-M2.7 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/minimax~1MiniMax-M2.7` | | | | | | | | 29 | [Muse Spark 1.3](https://benchize.com/models/muse-spark-1-3-xhigh.md)Extra high | 75% | SE ±0 pp | $14.76 | 19,082.4source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 14.757295 \| USD \| per_test \| Lower is better \| \| `latency` \| 19082.404 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131072 provider Meta reasoning Not reported Reasoning effort xhigh Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label meta/muse_spark_1_3 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/meta~1muse_spark_1_3` | | | | | | | | 29 | [Muse Spark 1.1](https://benchize.com/models/muse-spark-1-1-xhigh.md)Extra high | 75% | SE ±0 pp | $0.17 | 13,901.17source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.168085 \| USD \| per_test \| Lower is better \| \| `latency` \| 13901.169 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 256000 provider Meta reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label meta/muse_spark_1_1 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/meta~1muse_spark_1_1` | | | | | | | | 29 | [DeepSeek V4 Pro 0813](https://benchize.com/models/deepseek-v4-pro-0813-max.md)Max | 75% | SE ±14.43 pp | $0.1 | 19,187.82source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.09918 \| USD \| per_test \| Lower is better \| \| `latency` \| 19187.816 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 384000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±14.43 pp sourcePopulation 76 stderr 14.434 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label deepseek/deepseek-v4-pro-0813 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/deepseek~1deepseek-v4-pro-0813` | | | | | | | | 29 | [MiMo-V2.6-Pro](https://benchize.com/models/mimo-v2-6-pro.md) | 75% | SE ±0 pp | $1.62 | 20,895.49source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.6166 \| USD \| per_test \| Lower is better \| \| `latency` \| 20895.493 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Xiaomi reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label xiaomi/mimo-v2.6-pro Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/xiaomi~1mimo-v2.6-pro` | | | | | | | | 37 | [Gemini 3.1 Flash Lite Preview](https://benchize.com/models/gemini-3-1-flash-lite-preview-high.md)High | 66.67% | SE ±8.33 pp | $1.89 | 6,193.12source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.888892 \| USD \| per_test \| Lower is better \| \| `latency` \| 6193.118 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3.1-flash-lite-preview Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3.1-flash-lite-preview` | | | | | | | | 37 | [Gemini 3.5 Flash-Lite](https://benchize.com/models/gemini-3-5-flash-lite-high.md)High | 66.67% | SE ±8.33 pp | $7.5 | 9,264.53source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.499411 \| USD \| per_test \| Lower is better \| \| `latency` \| 9264.533 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3.5-flash-lite` | | | | | | | | 37 | [Mercury 2.5](https://benchize.com/models/mercury-2-5-high.md)High | 66.67% | SE ±8.33 pp | $4.78 | 15,328.54source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.783458 \| USD \| per_test \| Lower is better \| \| `latency` \| 15328.538 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Inception reasoning Not reported Reasoning effort high Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label inception/mercury-2.5 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/inception~1mercury-2.5` | | | | | | | | 37 | [DeepSeek V4 Pro](https://benchize.com/models/deepseek-v4-pro-off.md)Non-thinking | 66.67% | SE ±8.33 pp | $5.32 | 23,989.01source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.322002 \| USD \| per_test \| Lower is better \| \| `latency` \| 23989.013 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider DeepSeek reasoning false Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label deepseek/deepseek-v4-pro Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/deepseek~1deepseek-v4-pro` | | | | | | | | 37 | [Mistral Medium 3.5](https://benchize.com/models/mistral-medium-3-5-high.md)High | 66.67% | SE ±8.33 pp | $26.5 | 22,009.01source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 26.499387 \| USD \| per_test \| Lower is better \| \| `latency` \| 22009.013 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider Mistral AI reasoning Not reported Reasoning effort high Temperature 0.7 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label mistralai/mistral-medium-3.5 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/mistralai~1mistral-medium-3.5` | | | | | | | | 42 | [Laguna Xs 2](https://benchize.com/models/laguna-xs-2.md) | 58.33% | SE ±8.33 pp | $5.65 | 27,113.35source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 58.333 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.645149 \| USD \| per_test \| Lower is better \| \| `latency` \| 27113.353 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 32768 provider Poolside reasoning Not reported Reasoning effort unknown Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label poolside/laguna-xs.2 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/poolside~1laguna-xs.2` | | | | | | | | 43 | [Command A Plus](https://benchize.com/models/command-a-plus.md) | 50% | SE ±0 pp | — | 0source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 50 \| percent \| reported \| Higher is better \| \| `latency` \| 0 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 64000 provider Cohere reasoning Not reported Reasoning effort unknown Temperature 0.9 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 76 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label cohere/command-a-plus-05-2026 Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/cohere~1command-a-plus-05-2026` | | | | | | | --- Cite as: Terminal-Bench 2.1 / easy — Benchize, https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Aeasy (retrieved 2026-10-21)