# Gemini 3.1 Pro PreviewHigh > Source: Benchize (https://benchize.com) — https://benchize.com/models/gemini-3-1-pro-preview-high. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/models/gemini-3-1-pro-preview-high. Google Canonical URL: https://benchize.com/models/gemini-3-1-pro-preview-high Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Google ## Benchmarks | Benchmark | Score | ± | Rank | Cost | Latency | Source | | | --- | --- | --- | --- | --- | --- | --- | --- | | [MMLU Pro / health](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Ahealth.md) | 86.43% | SE ±1.2 pp | 3 / 40 | $0.02 | 19.71source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 86.43 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.018611 \| USD \| per_test \| Lower is better \| \| `latency` \| 19.713 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65535 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.2 pp sourcePopulation 138 stderr 1.197 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/health/google~1gemini-3.1-pro-preview` | | | | | | | | | [SAGE / PDEs](https://benchize.com/benchmarks/vals-ai%3Asage%3APDEs.md) | 39.68% | SE ±6.15 pp | 21 / 41 | $0.07 | 63.77source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 39.683 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.072766 \| USD \| per_test \| Lower is better \| \| `latency` \| 63.765 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 30000 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.15 pp sourcePopulation 91 stderr 6.148 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:07.515Z Evidence SHA-256 `cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c` Source locator `/tasks/PDEs/google~1gemini-3.1-pro-preview` | | | | | | | | | [Vals Science Index / mysterymechanism](https://benchize.com/benchmarks/vals-ai%3Ascience_index%3Amysterymechanism.md) | 20.72% | SE ±2.73 pp | 14 / 17 | $1.07 | 360.46source duration | [Vals AI · 1.0](https://www.vals.ai/_astro/benchmark_view_science_index.CN7FN-f1.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 20.721 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.065147 \| USD \| per_test \| Lower is better \| \| `latency` \| 360.456 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.73 pp sourcePopulation 21 stderr 2.726 tokenTotals {"cache_read_tokens":214402702,"cache_write_tokens":0,"input_tokens":262123547,"output_tokens":8178370,"reasoning_tokens":3546375} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_science_index.CN7FN-f1.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1.0 Observed 2026-10-21T16:45:50.879Z Evidence SHA-256 `fa4313008e91752fcfc185aaa78665f780d0fc23d7c7198a9853dbf2eaaaef1f` Source locator `/tasks/mysterymechanism/google~1gemini-3.1-pro-preview` | | | | | | | | | [MirrorCode](https://benchize.com/benchmarks/epoch-ai%3Amirrorcode.md) | 8.89% | SE ±4.56 pp | 8 / 8 | — | — | [epoch ai · MirrorCode](https://epoch.ai/data/benchmark_data.zip#file=mirrorcode.csv) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 0.08888888888888888 \| score_fraction \| Best score (across scorers) \| Higher is better \| \| `mean_score` \| 0.08888888888888888 \| score_fraction \| mean_score \| Higher is better \| ### Configuration benchmarkVersion MirrorCode reasoningBudget Not reported Reasoning effort high sourceModelVersion gemini-3.1-pro-preview_high ### Reported statistics Uncertainty Standard error: ±4.56 pp standardError 0.04561939004190448 ### Source [Original source](https://epoch.ai/data/benchmark_data.zip#file=mirrorcode.csv) Source epoch ai Original model label gemini-3.1-pro-preview Source version MirrorCode Evaluated 2026-08-10T16:12:46.343Z Observed 2026-10-21T16:30:24.199Z Evidence SHA-256 `5c31ffdeca24dd23e8d1dc40da67f2ec3e2e6f6d63e1a2b2791239e29f5b454e` Source locator `/rows/4` | | | | | | | | | [MedQA / overall](https://benchize.com/benchmarks/vals-ai%3Amedqa%3Aoverall.md) | 96.37% | SE ±0.17 pp | 1 / 12 | $0.02 | 38.67source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_medqa.Cco1kvog.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 96.367 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.017689 \| USD \| per_test \| Lower is better \| \| `latency` \| 38.67 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.17 pp sourcePopulation 95 stderr 0.171 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_medqa.Cco1kvog.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:00.691Z Evidence SHA-256 `d67905978d932f64698ebe9e230d9519e5bae1c9d7176dda8379dc3874b6f90f` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [AIME / aime 2024](https://benchize.com/benchmarks/vals-ai%3Aaime%3Aaime_2024.md) | 99.17% | SE ±0.55 pp | 1 / 14 | $0.11 | 93.32source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_aime.C3_jItdR.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 99.167 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.111396 \| USD \| per_test \| Lower is better \| \| `latency` \| 93.324 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.55 pp sourcePopulation 96 stderr 0.546 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_aime.C3_jItdR.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:01.920Z Evidence SHA-256 `7d016210b39d24351bbb5d12be09a973decac461bb2486eb1f23724fed2b7e6c` Source locator `/tasks/aime_2024/google~1gemini-3.1-pro-preview` | | | | | | | | | [SRE Bench / solved](https://benchize.com/benchmarks/vals-ai%3Asrebench%3Asolved.md) | 0% | SE ±0 pp | 23 / 24 | $11.34 | 3,803.74source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_sre_bench.Bd0uBJxH.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 11.336961 \| USD \| per_test \| Lower is better \| \| `latency` \| 3803.735 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 35 stderr 0 tokenTotals {"cache_read_tokens":5490229595,"cache_write_tokens":null,"input_tokens":5876432996,"output_tokens":22062729,"reasoning_tokens":15158240} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_sre_bench.Bd0uBJxH.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:15.724Z Evidence SHA-256 `a10b2c964f5878b0fc21c888703f311a5ef7d7a74fc3c529cab3ddd9fa474e2b` Source locator `/tasks/solved/google~1gemini-3.1-pro-preview` | | | | | | | | | [ProgramBench / strict](https://benchize.com/benchmarks/vals-ai%3Aprogrambench%3Astrict.md) | 0% | SE ±0 pp | 23 / 40 | $2.63 | 755.93source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.629095 \| USD \| per_test \| Lower is better \| \| `latency` \| 755.928 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 64 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:10.137Z Evidence SHA-256 `e9bbb4378c7a058441a21d456baf9316b96c2cf5b58d371a3c7c803f5000bd8c` Source locator `/tasks/strict/google~1gemini-3.1-pro-preview` | | | | | | | | | [MMLU Pro / math](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Amath.md) | 96% | SE ±0.53 pp | 6 / 40 | $0.03 | 24.46source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 96.003 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.026993 \| USD \| per_test \| Lower is better \| \| `latency` \| 24.464 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65535 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.53 pp sourcePopulation 138 stderr 0.533 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/math/google~1gemini-3.1-pro-preview` | | | | | | | | | [OTIS Mock AIME 2024–2025](https://benchize.com/benchmarks/epoch-ai%3Aotis_mock_aime_2024_2025.md) | 95.56% | SE ±3.11 pp | 28 / 123 | — | — | [epoch ai · OTIS Mock AIME 2024-2025](https://epoch.ai/data/benchmark_data.zip#file=otis_mock_aime_2024_2025.csv) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 0.9555555555555556 \| score_fraction \| Best score (across scorers) \| Higher is better \| \| `mean_score` \| 0.9555555555555556 \| score_fraction \| mean_score \| Higher is better \| ### Configuration benchmarkVersion OTIS Mock AIME 2024-2025 reasoningBudget Not reported Reasoning effort high sourceModelVersion gemini-3.1-pro-preview_high ### Reported statistics Uncertainty Standard error: ±3.11 pp standardError 0.031067790907534736 ### Source [Original source](https://epoch.ai/data/benchmark_data.zip#file=otis_mock_aime_2024_2025.csv) Source epoch ai Original model label gemini-3.1-pro-preview Source version OTIS Mock AIME 2024-2025 Evaluated 2026-08-06T23:59:11.000Z Observed 2026-10-21T16:30:24.199Z Evidence SHA-256 `5b411612f91e155dff4b09d09b548b42f7936482ea8b3c00b510bd46e2f6bc2c` Source locator `/rows/116` | | | | | | | | | [MedQA / asian](https://benchize.com/benchmarks/vals-ai%3Amedqa%3Aasian.md) | 96.3% | SE ±0.42 pp | 1 / 12 | $0.02 | 38.16source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_medqa.Cco1kvog.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 96.3 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.017279 \| USD \| per_test \| Lower is better \| \| `latency` \| 38.157 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.42 pp sourcePopulation 95 stderr 0.422 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_medqa.Cco1kvog.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:00.691Z Evidence SHA-256 `d67905978d932f64698ebe9e230d9519e5bae1c9d7176dda8379dc3874b6f90f` Source locator `/tasks/asian/google~1gemini-3.1-pro-preview` | | | | | | | | | [Code Migration / cli](https://benchize.com/benchmarks/vals-ai%3Acode-migration%3Acli.md) | 10.83% | SE ±1.47 pp | 38 / 47 | $1.52 | 735.95source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 10.833 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.521038 \| USD \| per_test \| Lower is better \| \| `latency` \| 735.951 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.47 pp sourcePopulation 79 stderr 1.468 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:12.712Z Evidence SHA-256 `99194597f2c76c402195be47839409046e18bd30454616da6dd40b736a088fa4` Source locator `/tasks/cli/google~1gemini-3.1-pro-preview` | | | | | | | | | [Public Benefits Bench v1 / overall](https://benchize.com/benchmarks/vals-ai%3Apublic-benefits-bench-v1%3Aoverall.md) | 50.88% | SE ±1.3 pp | 8 / 8 | $0.3 | 177.91source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_snap_v1.BPa0Hrre.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 50.88 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.300864 \| USD \| per_test \| Lower is better \| \| `latency` \| 177.913 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.3 pp sourcePopulation 13 stderr 1.3 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_snap_v1.BPa0Hrre.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:18.363Z Evidence SHA-256 `b85e2b3cbdf03e13e2ffdb27ce4f108003a548d705a24ae26a0a115e2027475d` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [AIME / overall](https://benchize.com/benchmarks/vals-ai%3Aaime%3Aoverall.md) | 98.13% | SE ±0.34 pp | 1 / 14 | $0.13 | 104.53source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_aime.C3_jItdR.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 98.125 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.126991 \| USD \| per_test \| Lower is better \| \| `latency` \| 104.525 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.34 pp sourcePopulation 96 stderr 0.343 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_aime.C3_jItdR.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:01.920Z Evidence SHA-256 `7d016210b39d24351bbb5d12be09a973decac461bb2486eb1f23724fed2b7e6c` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [Finance Agent (v2) / adjustments](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Aadjustments.md) | 38.36% | SE ±1.03 pp | 39 / 50 | $1.42 | 188.41source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 38.361 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.417992 \| USD \| per_test \| Lower is better \| \| `latency` \| 188.406 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.03 pp sourcePopulation 80 stderr 1.026 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:45:57.035Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/adjustments/google~1gemini-3.1-pro-preview` | | | | | | | | | [LiveCodeBench / overall](https://benchize.com/benchmarks/vals-ai%3Alcb%3Aoverall.md) | 88.49% | SE ±0.93 pp | 6 / 43 | $0.1 | 88.89source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 88.485 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.101147 \| USD \| per_test \| Lower is better \| \| `latency` \| 88.891 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.93 pp sourcePopulation 143 stderr 0.931 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:14.220Z Evidence SHA-256 `1a7c7dabc9d95a0182886b1edc42623987cac2095a0d407a5e334f37acdfa2f5` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [Public Benefits Bench v1 / both](https://benchize.com/benchmarks/vals-ai%3Apublic-benefits-bench-v1%3Aboth.md) | 50.88% | SE ±1.3 pp | 8 / 8 | $0.3 | 177.91source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_snap_v1.BPa0Hrre.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 50.88 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.300864 \| USD \| per_test \| Lower is better \| \| `latency` \| 177.913 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.3 pp sourcePopulation 13 stderr 1.3 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_snap_v1.BPa0Hrre.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:18.363Z Evidence SHA-256 `b85e2b3cbdf03e13e2ffdb27ce4f108003a548d705a24ae26a0a115e2027475d` Source locator `/tasks/both/google~1gemini-3.1-pro-preview` | | | | | | | | | [MMLU Pro / other](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aother.md) | 88.64% | SE ±1.04 pp | 3 / 40 | $0.02 | 15.07source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 88.636 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.016267 \| USD \| per_test \| Lower is better \| \| `latency` \| 15.066 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65535 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.04 pp sourcePopulation 138 stderr 1.044 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/other/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / corporate governance](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acorporate_governance.md) | 0% | SE ±0 pp | 11 / 49 | $0.63 | 445.24source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.63414 \| USD \| per_test \| Lower is better \| \| `latency` \| 445.236 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/corporate_governance/google~1gemini-3.1-pro-preview` | | | | | | | | | [Tax Agent Bench / fact pattern analysis](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Afact_pattern_analysis.md) | 6.98% | SE ±3.93 pp | 41 / 46 | $0.27 | 115.16source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 6.977 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.274484 \| USD \| per_test \| Lower is better \| \| `latency` \| 115.164 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.93 pp sourcePopulation 73 stderr 3.931 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/fact_pattern_analysis/google~1gemini-3.1-pro-preview` | | | | | | | | | [Tax Agent Bench / weighted pass current temporal analysis](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Aweighted_pass__current_temporal_analysis.md) | 36.54% | SE ±7.94 pp | 39 / 46 | $0.31 | 138.83source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 36.539 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.312752 \| USD \| per_test \| Lower is better \| \| `latency` \| 138.829 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.94 pp sourcePopulation 73 stderr 7.942 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/weighted_pass__current_temporal_analysis/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / immigration](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aimmigration.md) | 0% | SE ±0 pp | 6 / 49 | $0.75 | 298.58source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.74639 \| USD \| per_test \| Lower is better \| \| `latency` \| 298.584 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/immigration/google~1gemini-3.1-pro-preview` | | | | | | | | | [EMB / comps](https://benchize.com/benchmarks/vals-ai%3Aemb%3Acomps.md) | 44.51% | SE ±7.62 pp | 37 / 47 | $7.47 | 768.42source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 44.505 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.466867 \| USD \| per_test \| Lower is better \| \| `latency` \| 768.417 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.62 pp sourcePopulation 76 stderr 7.615 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3.1-pro-preview` | | | | | | | | | [SimpleQA Verified](https://benchize.com/benchmarks/epoch-ai%3Asimpleqa_verified.md) | 73.5% | SE ±1.4 pp | 4 / 39 | — | — | [epoch ai · SimpleQA Verified](https://epoch.ai/data/benchmark_data.zip#file=simpleqa_verified.csv) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 0.735 \| score_fraction \| Best score (across scorers) \| Higher is better \| \| `mean_score` \| 0.735 \| score_fraction \| mean_score \| Higher is better \| ### Configuration benchmarkVersion SimpleQA Verified reasoningBudget Not reported Reasoning effort high sourceModelVersion gemini-3.1-pro-preview_high ### Reported statistics Uncertainty Standard error: ±1.4 pp standardError 0.013963164754810065 ### Source [Original source](https://epoch.ai/data/benchmark_data.zip#file=simpleqa_verified.csv) Source epoch ai Original model label gemini-3.1-pro-preview Source version SimpleQA Verified Evaluated 2026-08-10T21:35:39.000Z Observed 2026-10-21T16:30:24.199Z Evidence SHA-256 `d4ef3a7d6ccc9c611a30e98d0561021e903c0f4a224851311a7c420862154dd8` Source locator `/rows/86` | | | | | | | | | [Terminal-Bench 2.0 / overall](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2%3Aoverall.md) | 67.42% | SE ±5 pp | 4 / 23 | $0.52 | 542.89source duration | [Vals AI · 2.0](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2.T48tPR4O.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 67.416 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.516084 \| USD \| per_test \| Lower is better \| \| `latency` \| 542.887 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5 pp sourcePopulation 67 stderr 4.996 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2.T48tPR4O.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2.0 Observed 2026-10-21T16:46:14.732Z Evidence SHA-256 `38e640d995259778e531f048b2f2c7b3b281cb05b56be643ddc9dc53eb85adf7` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate environmental esg](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__environmental_esg.md) | 58.78% | SE ±2.95 pp | 43 / 49 | $0.53 | 223.29source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 58.781 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.526047 \| USD \| per_test \| Lower is better \| \| `latency` \| 223.294 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.95 pp sourcePopulation 81 stderr 2.952 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__environmental_esg/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / tax](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Atax.md) | 0% | SE ±0 pp | 3 / 49 | $2.24 | 547.97source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.239645 \| USD \| per_test \| Lower is better \| \| `latency` \| 547.97 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/tax/google~1gemini-3.1-pro-preview` | | | | | | | | | [Legal Research Bench / administrative regulatory](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aadministrative_regulatory.md) | 24.24% | SE ±5.32 pp | 40 / 50 | $1.23 | 219.63source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 24.242 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.225685 \| USD \| per_test \| Lower is better \| \| `latency` \| 219.629 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.32 pp sourcePopulation 82 stderr 5.316 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/administrative_regulatory/google~1gemini-3.1-pro-preview` | | | | | | | | | [CorpFin v2 / shared max context](https://benchize.com/benchmarks/vals-ai%3Acorp_fin_v2%3Ashared_max_context.md) | 65.73% | SE ±1.62 pp | 15 / 39 | $0.15 | 28.14source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.734 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.145587 \| USD \| per_test \| Lower is better \| \| `latency` \| 28.137 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.62 pp sourcePopulation 134 stderr 1.62 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:45:57.998Z Evidence SHA-256 `7cf2217ab3728d5a84bffce82aab9fc52d7f810b0cbe8c65b387df47f3d31c9b` Source locator `/tasks/shared_max_context/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / structured finance securitization](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Astructured_finance_securitization.md) | 0% | SE ±0 pp | 12 / 49 | $2.81 | 663.88source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.811066 \| USD \| per_test \| Lower is better \| \| `latency` \| 663.878 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/structured_finance_securitization/google~1gemini-3.1-pro-preview` | | | | | | | | | [Legal Research Bench / weighted pass constitutional civil rights](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aweighted_pass__constitutional_civil_rights.md) | 79.64% | SE ±2.87 pp | 37 / 50 | $0.47 | 223.21source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 79.637 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.466165 \| USD \| per_test \| Lower is better \| \| `latency` \| 223.214 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.87 pp sourcePopulation 82 stderr 2.872 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/weighted_pass__constitutional_civil_rights/google~1gemini-3.1-pro-preview` | | | | | | | | | [SAGE / automata](https://benchize.com/benchmarks/vals-ai%3Asage%3Aautomata.md) | 32.14% | SE ±5.94 pp | 34 / 41 | $0.07 | 62.43source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 32.143 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.068462 \| USD \| per_test \| Lower is better \| \| `latency` \| 62.434 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 30000 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.94 pp sourcePopulation 91 stderr 5.937 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:07.515Z Evidence SHA-256 `cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c` Source locator `/tasks/automata/google~1gemini-3.1-pro-preview` | | | | | | | | | [Terminal-Bench 4.0 / software](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-4%3Asoftware.md) | 3.7% | SE ±1.85 pp | 32 / 36 | — | — | [Vals AI · 4.0](https://www.vals.ai/_astro/benchmark_view_terminal_bench_4.CAnGyzC_.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 3.704 \| percent \| reported \| Higher is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.85 pp sourcePopulation 52 stderr 1.852 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_4.CAnGyzC_.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 4.0 Observed 2026-10-21T16:46:10.642Z Evidence SHA-256 `98f6964580e901f1aa143db664bd92aa558415f643569492545d67c47d7f519b` Source locator `/tasks/software/google~1gemini-3.1-pro-preview` | | | | | | | | | [EMB / lbo](https://benchize.com/benchmarks/vals-ai%3Aemb%3Albo.md) | 37.97% | SE ±4.01 pp | 40 / 47 | $4.76 | 784.35source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 37.967 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.762336 \| USD \| per_test \| Lower is better \| \| `latency` \| 784.347 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.01 pp sourcePopulation 76 stderr 4.006 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/lbo/google~1gemini-3.1-pro-preview` | | | | | | | | | [LiveCodeBench / easy](https://benchize.com/benchmarks/vals-ai%3Alcb%3Aeasy.md) | 99.07% | SE ±0.54 pp | 1 / 43 | $0.02 | 19.29source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 99.068 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.021104 \| USD \| per_test \| Lower is better \| \| `latency` \| 19.291 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.54 pp sourcePopulation 143 stderr 0.535 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:14.220Z Evidence SHA-256 `1a7c7dabc9d95a0182886b1edc42623987cac2095a0d407a5e334f37acdfa2f5` Source locator `/tasks/easy/google~1gemini-3.1-pro-preview` | | | | | | | | | [Legal Research Bench / overall](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aoverall.md) | 20.67% | SE ±2.82 pp | 39 / 50 | $1.1 | 242.05source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 20.673 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.100145 \| USD \| per_test \| Lower is better \| \| `latency` \| 242.05 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.82 pp sourcePopulation 82 stderr 2.815 tokenTotals {"cache_read_tokens":321431187,"cache_write_tokens":0,"input_tokens":354575717,"output_tokens":2983139,"reasoning_tokens":2206660} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [SAGE / overall](https://benchize.com/benchmarks/vals-ai%3Asage%3Aoverall.md) | 48.68% | SE ±3.29 pp | 17 / 41 | $0.07 | 61.17source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 48.677 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.068908 \| USD \| per_test \| Lower is better \| \| `latency` \| 61.174 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 30000 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.29 pp sourcePopulation 91 stderr 3.293 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:07.515Z Evidence SHA-256 `cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [BioMysteryBench / overall](https://benchize.com/benchmarks/vals-ai%3Abiomysterybench%3Aoverall.md) | 60% | SE ±2.94 pp | 19 / 20 | $0.71 | 251.87source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_biomysterybench.Bley-WAP.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 60 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.713378 \| USD \| per_test \| Lower is better \| \| `latency` \| 251.867 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Terminus 2 maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.94 pp sourcePopulation 27 stderr 2.94 tokenTotals {"cache_read_tokens":62626251,"cache_write_tokens":0,"input_tokens":74631111,"output_tokens":2138194,"reasoning_tokens":1319799} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_biomysterybench.Bley-WAP.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:04.029Z Evidence SHA-256 `59bf79d38ec415a34ffb1c2901a16ecbbe101b7fa864e86104d2571fc7cfc23d` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [Legal Research Bench / weighted pass civil litigation](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aweighted_pass__civil_litigation.md) | 82.71% | SE ±2.97 pp | 35 / 50 | $0.51 | 225.62source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.707 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.508937 \| USD \| per_test \| Lower is better \| \| `latency` \| 225.619 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.97 pp sourcePopulation 82 stderr 2.966 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/weighted_pass__civil_litigation/google~1gemini-3.1-pro-preview` | | | | | | | | | [Vals Science Index / terminal bench science](https://benchize.com/benchmarks/vals-ai%3Ascience_index%3Aterminal_bench_science.md) | 1.43% | SE ±1.43 pp | 15 / 17 | $5.91 | 2,638.07source duration | [Vals AI · 1.0](https://www.vals.ai/_astro/benchmark_view_science_index.CN7FN-f1.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 1.429 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 5.907095 \| USD \| per_test \| Lower is better \| \| `latency` \| 2638.065 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.43 pp sourcePopulation 21 stderr 1.429 tokenTotals {"cache_read_tokens":796983036,"cache_write_tokens":0,"input_tokens":1655712288,"output_tokens":6012954,"reasoning_tokens":2054169} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_science_index.CN7FN-f1.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1.0 Observed 2026-10-21T16:45:50.879Z Evidence SHA-256 `fa4313008e91752fcfc185aaa78665f780d0fc23d7c7198a9853dbf2eaaaef1f` Source locator `/tasks/terminal_bench_science/google~1gemini-3.1-pro-preview` | | | | | | | | | [ProgramBench / overall](https://benchize.com/benchmarks/vals-ai%3Aprogrambench%3Aoverall.md) | 0% | SE ±0 pp | 23 / 40 | $2.63 | 755.93source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.629095 \| USD \| per_test \| Lower is better \| \| `latency` \| 755.928 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 64 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:10.137Z Evidence SHA-256 `e9bbb4378c7a058441a21d456baf9316b96c2cf5b58d371a3c7c803f5000bd8c` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [CorpFin v2 / max fitting context](https://benchize.com/benchmarks/vals-ai%3Acorp_fin_v2%3Amax_fitting_context.md) | 66.9% | SE ±1.61 pp | 11 / 39 | $0.39 | 28.94source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.9 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.387473 \| USD \| per_test \| Lower is better \| \| `latency` \| 28.935 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.61 pp sourcePopulation 134 stderr 1.607 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:45:57.998Z Evidence SHA-256 `7cf2217ab3728d5a84bffce82aab9fc52d7f810b0cbe8c65b387df47f3d31c9b` Source locator `/tasks/max_fitting_context/google~1gemini-3.1-pro-preview` | | | | | | | | | [SWE-bench / overall](https://benchize.com/benchmarks/vals-ai%3Aswebench%3Aoverall.md) | 78.8% | SE ±1.83 pp | 17 / 39 | $0.78 | 312.26source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 78.8 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.783381 \| USD \| per_test \| Lower is better \| \| `latency` \| 312.261 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.83 pp sourcePopulation 88 stderr 1.83 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:13.711Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [SWE-bench / <15 min fix](https://benchize.com/benchmarks/vals-ai%3Aswebench%3A%3C15%20min%20fix.md) | 88.66% | SE ±2.28 pp | 14 / 39 | $0.65 | 259.78source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 88.66 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.652263 \| USD \| per_test \| Lower is better \| \| `latency` \| 259.783 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.28 pp sourcePopulation 88 stderr 2.282 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:13.711Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/<15 min fix/google~1gemini-3.1-pro-preview` | | | | | | | | | [TaxEval v2 / stepwise reasoning](https://benchize.com/benchmarks/vals-ai%3Atax_eval_v2%3Astepwise_reasoning.md) | 85.02% | SE ±1.02 pp | 18 / 40 | $0.03 | 41.93source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.016 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.03149 \| USD \| per_test \| Lower is better \| \| `latency` \| 41.931 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65535 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.02 pp sourcePopulation 145 stderr 1.019 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:45:59.000Z Evidence SHA-256 `8b354a6ae848f5bb25a2e8b061b5f576d5b9e0c11b941dc1e6bed078e6529cd6` Source locator `/tasks/stepwise_reasoning/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate immigration](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__immigration.md) | 54.91% | SE ±3.01 pp | 44 / 49 | $0.75 | 298.58source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 54.909 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.74639 \| USD \| per_test \| Lower is better \| \| `latency` \| 298.584 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.01 pp sourcePopulation 81 stderr 3.006 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__immigration/google~1gemini-3.1-pro-preview` | | | | | | | | | [EMB / ma](https://benchize.com/benchmarks/vals-ai%3Aemb%3Ama.md) | 58.83% | SE ±7.83 pp | 33 / 47 | $6.07 | 646.47source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 58.833 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 6.070452 \| USD \| per_test \| Lower is better \| \| `latency` \| 646.47 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.83 pp sourcePopulation 76 stderr 7.829 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate employment labor](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__employment_labor.md) | 49.06% | SE ±3.07 pp | 46 / 49 | $0.81 | 378.9source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 49.064 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.810734 \| USD \| per_test \| Lower is better \| \| `latency` \| 378.896 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.07 pp sourcePopulation 81 stderr 3.065 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__employment_labor/google~1gemini-3.1-pro-preview` | | | | | | | | | [Vibe Code Bench v1.1 / overall](https://benchize.com/benchmarks/vals-ai%3Avibe-code%3Aoverall.md) | 32.03% | SE ±4.34 pp | 46 / 59 | $3.83 | 1,211.91source duration | [Vals AI · 1.1](https://www.vals.ai/_astro/benchmark_view_vibe_code_bench.DcEBMbCA.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 32.034 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.825202 \| USD \| per_test \| Lower is better \| \| `latency` \| 1211.906 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness OpenHands maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.34 pp sourcePopulation 116 stderr 4.342 tokenTotals {"cache_read_tokens":678233584,"cache_write_tokens":0,"input_tokens":722761650,"output_tokens":1941198,"reasoning_tokens":260725} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vibe_code_bench.DcEBMbCA.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1.1 Observed 2026-10-21T16:46:08.625Z Evidence SHA-256 `0e8c23f521a0f751c5592952620e197b376cade41c78578461b7aa5740a173e7` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / energy natural resources](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aenergy_natural_resources.md) | 0% | SE ±0 pp | 28 / 49 | $1.92 | 350.94source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.916498 \| USD \| per_test \| Lower is better \| \| `latency` \| 350.94 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/energy_natural_resources/google~1gemini-3.1-pro-preview` | | | | | | | | | [IOI / ioi2024](https://benchize.com/benchmarks/vals-ai%3Aioi%3Aioi2024.md) | 55.5% | SE ±18.71 pp | 18 / 29 | $1.21 | 585.6source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_ioi.CuUolFNv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 55.5 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.207234 \| USD \| per_test \| Lower is better \| \| `latency` \| 585.596 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±18.71 pp sourcePopulation 44 stderr 18.714 tokenTotals {"cache_read_tokens":5955720,"cache_write_tokens":0,"input_tokens":7142113,"output_tokens":306623,"reasoning_tokens":243010} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_ioi.CuUolFNv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:46:11.757Z Evidence SHA-256 `e1545b21b5b5c98d037aefa9d441b65859d07a10550f242d0e16162a49b8e8ab` Source locator `/tasks/ioi2024/google~1gemini-3.1-pro-preview` | | | | | | | | | [Mystery Game Puzzles](https://benchize.com/benchmarks/epoch-ai%3Amystery_game_puzzles.md) | 34% | SE ±4.76 pp | 21 / 71 | — | — | [epoch ai · Mystery Game Puzzles](https://epoch.ai/data/benchmark_data.zip#file=mystery_game_puzzles.csv) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 0.34 \| score_fraction \| Best score (across scorers) \| Higher is better \| \| `mean_score` \| 0.34 \| score_fraction \| mean_score \| Higher is better \| ### Configuration benchmarkVersion Mystery Game Puzzles reasoningBudget Not reported Reasoning effort high sourceModelVersion gemini-3.1-pro-preview_high ### Reported statistics Uncertainty Standard error: ±4.76 pp standardError 0.04760952285695234 ### Source [Original source](https://epoch.ai/data/benchmark_data.zip#file=mystery_game_puzzles.csv) Source epoch ai Original model label gemini-3.1-pro-preview Source version Mystery Game Puzzles Evaluated 2026-07-27T18:29:16.000Z Observed 2026-10-21T16:30:24.199Z Evidence SHA-256 `795b977bc82c460a5e2c1d6dd7b2569ddb21853f3829c4f6fdd16edfaf47bfcf` Source locator `/rows/127` | | | | | | | | | [SAGE / calculus](https://benchize.com/benchmarks/vals-ai%3Asage%3Acalculus.md) | 78.4% | SE ±5.22 pp | 9 / 36 | $0.07 | 57.52source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 78.395 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.070287 \| USD \| per_test \| Lower is better \| \| `latency` \| 57.524 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 30000 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.22 pp sourcePopulation 91 stderr 5.215 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:07.515Z Evidence SHA-256 `cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c` Source locator `/tasks/calculus/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate.md) | 56.09% | SE ±0.6 pp | 44 / 49 | $1.15 | 436.69source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 56.086 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.149757 \| USD \| per_test \| Lower is better \| \| `latency` \| 436.69 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.6 pp sourcePopulation 81 stderr 0.597 tokenTotals {"cache_read_tokens":204622317,"cache_write_tokens":null,"input_tokens":230732445,"output_tokens":1350000,"reasoning_tokens":503310} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate/google~1gemini-3.1-pro-preview` | | | | | | | | | [Legal Research Bench / civil litigation](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Acivil_litigation.md) | 11.11% | SE ±6.16 pp | 41 / 50 | $0.51 | 225.62source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 11.111 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.508937 \| USD \| per_test \| Lower is better \| \| `latency` \| 225.619 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.16 pp sourcePopulation 82 stderr 6.163 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/civil_litigation/google~1gemini-3.1-pro-preview` | | | | | | | | | [SWE-bench / >4 hours](https://benchize.com/benchmarks/vals-ai%3Aswebench%3A%3E4%20hours.md) | 33.33% | SE ±33.33 pp | 13 / 39 | $1.03 | 424.84source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 33.333 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.025173 \| USD \| per_test \| Lower is better \| \| `latency` \| 424.841 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±33.33 pp sourcePopulation 88 stderr 33.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:13.711Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/>4 hours/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate arbitration international dispute resolution](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__arbitration_international_dispute_resolution.md) | 54.12% | SE ±2.99 pp | 40 / 49 | $0.99 | 704.48source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 54.122 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.990325 \| USD \| per_test \| Lower is better \| \| `latency` \| 704.482 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.99 pp sourcePopulation 81 stderr 2.989 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__arbitration_international_dispute_resolution/google~1gemini-3.1-pro-preview` | | | | | | | | | [Finance Agent (v2) / disclosure analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Adisclosure_analysis.md) | 46.18% | SE ±1.86 pp | 42 / 50 | $1.7 | 191.85source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 46.181 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.695327 \| USD \| per_test \| Lower is better \| \| `latency` \| 191.853 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.86 pp sourcePopulation 80 stderr 1.857 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:45:57.035Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/disclosure_analysis/google~1gemini-3.1-pro-preview` | | | | | | | | | [Code Migration / cobol](https://benchize.com/benchmarks/vals-ai%3Acode-migration%3Acobol.md) | 36.75% | SE ±15.11 pp | 39 / 47 | $1.9 | 674.44source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 36.753 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.895374 \| USD \| per_test \| Lower is better \| \| `latency` \| 674.444 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±15.11 pp sourcePopulation 79 stderr 15.105 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:12.712Z Evidence SHA-256 `99194597f2c76c402195be47839409046e18bd30454616da6dd40b736a088fa4` Source locator `/tasks/cobol/google~1gemini-3.1-pro-preview` | | | | | | | | | [BioMysteryBench / human solvable](https://benchize.com/benchmarks/vals-ai%3Abiomysterybench%3Ahuman_solvable.md) | 67.58% | SE ±2.42 pp | 19 / 20 | $0.65 | 234.5source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_biomysterybench.Bley-WAP.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 67.58 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.650575 \| USD \| per_test \| Lower is better \| \| `latency` \| 234.501 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Terminus 2 maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.42 pp sourcePopulation 27 stderr 2.416 tokenTotals {"cache_read_tokens":44628391,"cache_write_tokens":0,"input_tokens":53571907,"output_tokens":1627137,"reasoning_tokens":1024131} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_biomysterybench.Bley-WAP.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:04.029Z Evidence SHA-256 `59bf79d38ec415a34ffb1c2901a16ecbbe101b7fa864e86104d2571fc7cfc23d` Source locator `/tasks/human_solvable/google~1gemini-3.1-pro-preview` | | | | | | | | | [MedQA / black](https://benchize.com/benchmarks/vals-ai%3Amedqa%3Ablack.md) | 96.8% | SE ±0.39 pp | 1 / 12 | $0.02 | 39.84source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_medqa.Cco1kvog.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 96.8 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.018058 \| USD \| per_test \| Lower is better \| \| `latency` \| 39.836 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.39 pp sourcePopulation 95 stderr 0.394 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_medqa.Cco1kvog.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:00.691Z Evidence SHA-256 `d67905978d932f64698ebe9e230d9519e5bae1c9d7176dda8379dc3874b6f90f` Source locator `/tasks/black/google~1gemini-3.1-pro-preview` | | | | | | | | | [CaseLaw v2 / overall](https://benchize.com/benchmarks/vals-ai%3Acase_law_v2%3Aoverall.md) | 64.85% | SE ±1.24 pp | 3 / 17 | $0.14 | 56.79source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.845 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.136185 \| USD \| per_test \| Lower is better \| \| `latency` \| 56.785 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.24 pp sourcePopulation 54 stderr 1.236 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [Terminal-Bench-Science / life sciences](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-science%3Alife_sciences.md) | 0% | SE ±0 pp | 24 / 30 | — | — | [Vals AI · 0.1](https://www.vals.ai/_astro/benchmark_view_terminal_bench_science.o-HR8Nyt.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| ### Configuration computeEffort Not reported harness Mini-SWE-agent maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 41 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_science.o-HR8Nyt.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 0.1 Observed 2026-10-21T16:46:03.529Z Evidence SHA-256 `a30a8618deaecf018ea1c8c9f675576cdd9ab79742d05788067db49afcc5f91b` Source locator `/tasks/life_sciences/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate emerging companies venture capital](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__emerging_companies_venture_capital.md) | 59.39% | SE ±2.87 pp | 43 / 49 | $1.64 | 536.08source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 59.386 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.644529 \| USD \| per_test \| Lower is better \| \| `latency` \| 536.078 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.87 pp sourcePopulation 81 stderr 2.874 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__emerging_companies_venture_capital/google~1gemini-3.1-pro-preview` | | | | | | | | | [MMLU Pro / psychology](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Apsychology.md) | 91.98% | SE ±0.96 pp | 4 / 40 | $0.02 | 15.97source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.98 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.015685 \| USD \| per_test \| Lower is better \| \| `latency` \| 15.965 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65535 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.96 pp sourcePopulation 138 stderr 0.961 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/psychology/google~1gemini-3.1-pro-preview` | | | | | | | | | [ProgramBench / partial](https://benchize.com/benchmarks/vals-ai%3Aprogrambench%3Apartial.md) | 39.47% | SE ±1.81 pp | 32 / 40 | $2.63 | 755.93source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 39.468 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.629095 \| USD \| per_test \| Lower is better \| \| `latency` \| 755.928 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.81 pp sourcePopulation 64 stderr 1.809 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:10.137Z Evidence SHA-256 `e9bbb4378c7a058441a21d456baf9316b96c2cf5b58d371a3c7c803f5000bd8c` Source locator `/tasks/partial/google~1gemini-3.1-pro-preview` | | | | | | | | | [SRE Bench / partial](https://benchize.com/benchmarks/vals-ai%3Asrebench%3Apartial.md) | 0.95% | SE ±0.34 pp | 24 / 24 | $11.34 | 3,803.74source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_sre_bench.Bd0uBJxH.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0.954 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 11.336961 \| USD \| per_test \| Lower is better \| \| `latency` \| 3803.735 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.34 pp sourcePopulation 35 stderr 0.338 tokenTotals {"cache_read_tokens":5490229595,"cache_write_tokens":null,"input_tokens":5876432996,"output_tokens":22062729,"reasoning_tokens":15158240} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_sre_bench.Bd0uBJxH.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:15.724Z Evidence SHA-256 `a10b2c964f5878b0fc21c888703f311a5ef7d7a74fc3c529cab3ddd9fa474e2b` Source locator `/tasks/partial/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / white collar defense investigations](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Awhite_collar_defense_investigations.md) | 0% | SE ±0 pp | 1 / 49 | $0.67 | 275.23source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.665432 \| USD \| per_test \| Lower is better \| \| `latency` \| 275.234 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/white_collar_defense_investigations/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / trusts estates private client](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Atrusts_estates_private_client.md) | 0% | SE ±0 pp | 15 / 49 | $0.53 | 484.97source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.526516 \| USD \| per_test \| Lower is better \| \| `latency` \| 484.966 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/trusts_estates_private_client/google~1gemini-3.1-pro-preview` | | | | | | | | | [Terminal-Bench-Science / earth sciences](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-science%3Aearth_sciences.md) | 0% | SE ±0 pp | 14 / 30 | — | — | [Vals AI · 0.1](https://www.vals.ai/_astro/benchmark_view_terminal_bench_science.o-HR8Nyt.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| ### Configuration computeEffort Not reported harness Mini-SWE-agent maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 41 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_science.o-HR8Nyt.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 0.1 Observed 2026-10-21T16:46:03.529Z Evidence SHA-256 `a30a8618deaecf018ea1c8c9f675576cdd9ab79742d05788067db49afcc5f91b` Source locator `/tasks/earth_sciences/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate antitrust competition](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__antitrust_competition.md) | 50.73% | SE ±3.02 pp | 44 / 49 | $1.01 | 515.76source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 50.725 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.014086 \| USD \| per_test \| Lower is better \| \| `latency` \| 515.756 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.02 pp sourcePopulation 81 stderr 3.015 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__antitrust_competition/google~1gemini-3.1-pro-preview` | | | | | | | | | [MMLU Pro / business](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Abusiness.md) | 93.41% | SE ±0.88 pp | 5 / 40 | $0.02 | 19.55source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 93.409 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.023771 \| USD \| per_test \| Lower is better \| \| `latency` \| 19.549 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65535 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.88 pp sourcePopulation 138 stderr 0.883 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/business/google~1gemini-3.1-pro-preview` | | | | | | | | | [GPQA Diamond / diamond zero shot cot](https://benchize.com/benchmarks/vals-ai%3Agpqa%3Adiamond_zero_shot_cot.md) | 94.95% | SE ±1.56 pp | 2 / 40 | $0.08 | 64.02source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_gpqa.ghZZBpyU.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 94.949 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.077857 \| USD \| per_test \| Lower is better \| \| `latency` \| 64.024 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65535 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.56 pp sourcePopulation 138 stderr 1.556 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_gpqa.ghZZBpyU.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:06.040Z Evidence SHA-256 `951eb5224097733a9d09d06c14908ae2980e2e640e539f2c284877c5092d7077` Source locator `/tasks/diamond_zero_shot_cot/google~1gemini-3.1-pro-preview` | | | | | | | | | [LegalBench / interpretation tasks](https://benchize.com/benchmarks/vals-ai%3Alegal_bench%3Ainterpretation_tasks.md) | 83.82% | SE ±0.37 pp | 12 / 44 | $0.00609 | 9.14source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.823 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.006094 \| USD \| per_test \| Lower is better \| \| `latency` \| 9.143 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.37 pp sourcePopulation 151 stderr 0.367 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.390Z Evidence SHA-256 `c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52` Source locator `/tasks/interpretation_tasks/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / antitrust competition](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aantitrust_competition.md) | 0% | SE ±0 pp | 7 / 49 | $1.01 | 515.76source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.014086 \| USD \| per_test \| Lower is better \| \| `latency` \| 515.756 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/antitrust_competition/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / international trade sanctions](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Ainternational_trade_sanctions.md) | 0% | SE ±0 pp | 4 / 49 | $0.59 | 551.72source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.590442 \| USD \| per_test \| Lower is better \| \| `latency` \| 551.724 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/international_trade_sanctions/google~1gemini-3.1-pro-preview` | | | | | | | | | [ToneBench Scriptwriting](https://benchize.com/benchmarks/tonebench%3Ascripts.md) | 1,171.9 | Range 1,126.1–1,218.8 | 91 / 126 | — | — | [tonebench](https://benchmark.towardsai.com/data/leaderboard.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `elo` \| 1171.9 \| rating \| reported \| Higher is better \| \| `overall` \| 70.99 \| percent \| reported \| Higher is better \| \| `anti_slop_numbers` \| 64.12 \| source_score \| rubric \| Higher is better \| \| `continuity_emotion` \| 72.63 \| source_score \| rubric \| Higher is better \| \| `cue_quality` \| 69.75 \| source_score \| rubric \| Higher is better \| \| `hook_strength` \| 71.85 \| source_score \| rubric \| Higher is better \| \| `length_adherence` \| 79.39 \| source_score \| rubric \| Higher is better \| \| `slop` \| 79.42 \| source_score \| rubric \| Higher is better \| \| `slop_score` \| 94.73 \| source_score \| rubric \| Higher is better \| \| `substance_accuracy` \| 73.87 \| source_score \| rubric \| Higher is better \| \| `tone_voice` \| 63.61 \| source_score \| rubric \| Higher is better \| \| `writing_quality` \| 68.12 \| source_score \| rubric \| Higher is better \| \| `youtube_best_practices` \| 70.85 \| source_score \| rubric \| Higher is better \| ### Configuration provider google Reasoning effort high scoreScope 10_task_current sourceConfiguration gemini-3-1-pro-high ### Reported statistics Uncertainty Reported interval: 1,126.1–1,218.8 interval [1126.1,1218.8] rank 94 runCount 5 ### Source [Original source](https://benchmark.towardsai.com/data/leaderboard.json) Source tonebench Original model label Gemini 3.1 Pro (high thinking) Evaluated 2026-10-11 Observed 2026-10-21T16:32:48.862Z Evidence SHA-256 `4264338337f418d3ded7ace250cad64fe12cacbd4b79a312bfaf0ae9268a01c3` Source locator `/rows/93` | | | | | | | | | [CorpFin v2 / exact pages](https://benchize.com/benchmarks/vals-ai%3Acorp_fin_v2%3Aexact_pages.md) | 60.84% | SE ±1.67 pp | 23 / 39 | $0.01 | 19.09source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 60.839 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.012811 \| USD \| per_test \| Lower is better \| \| `latency` \| 19.092 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.67 pp sourcePopulation 134 stderr 1.666 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:45:57.998Z Evidence SHA-256 `7cf2217ab3728d5a84bffce82aab9fc52d7f810b0cbe8c65b387df47f3d31c9b` Source locator `/tasks/exact_pages/google~1gemini-3.1-pro-preview` | | | | | | | | | [Terminal-Bench 2.1 / easy](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Aeasy.md) | 91.67% | SE ±8.33 pp | 15 / 43 | $12.96 | 9,128.05source duration | [Vals AI · 2.1](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 12.964574 \| USD \| per_test \| Lower is better \| \| `latency` \| 9128.048 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3.1-pro-preview` | | | | | | | | | [EMB / dcf](https://benchize.com/benchmarks/vals-ai%3Aemb%3Adcf.md) | 46.39% | SE ±7.72 pp | 34 / 47 | $2.5 | 584.59source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 46.388 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.495552 \| USD \| per_test \| Lower is better \| \| `latency` \| 584.588 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.72 pp sourcePopulation 76 stderr 7.715 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dcf/google~1gemini-3.1-pro-preview` | | | | | | | | | [SWE-bench / 1-4 hours](https://benchize.com/benchmarks/vals-ai%3Aswebench%3A1-4%20hours.md) | 42.86% | SE ±7.73 pp | 22 / 39 | $1 | 386.06source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 42.857 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.999095 \| USD \| per_test \| Lower is better \| \| `latency` \| 386.06 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.73 pp sourcePopulation 88 stderr 7.729 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:13.711Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/1-4 hours/google~1gemini-3.1-pro-preview` | | | | | | | | | [Furniture Assembly Benchmark](https://benchize.com/benchmarks/epoch-ai%3Afurniture_assembly.md) | 26.67% | SE ±5.63 pp | 18 / 21 | — | — | [epoch ai · Furniture Assembly](https://epoch.ai/data/benchmark_data.zip#file=furniture_assembly.csv) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 0.26666666666666666 \| score_fraction \| Best score (across scorers) \| Higher is better \| \| `mean_score` \| 0.26666666666666666 \| score_fraction \| mean_score \| Higher is better \| ### Configuration benchmarkVersion Furniture Assembly reasoningBudget Not reported Reasoning effort high sourceModelVersion gemini-3.1-pro-preview_high ### Reported statistics Uncertainty Standard error: ±5.63 pp standardError 0.056331678510110804 ### Source [Original source](https://epoch.ai/data/benchmark_data.zip#file=furniture_assembly.csv) Source epoch ai Original model label gemini-3.1-pro-preview Source version Furniture Assembly Evaluated 2026-09-10T20:23:45.000Z Observed 2026-10-21T16:30:24.199Z Evidence SHA-256 `d62ab28005a1b7ab0e57b227d04b0cf86c70a5d8a1981addb26d2ac41c116865` Source locator `/rows/20` | | | | | | | | | [MMLU Pro / engineering](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aengineering.md) | 88.03% | SE ±1.04 pp | 7 / 40 | $0.08 | 64.58source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 88.029 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.079443 \| USD \| per_test \| Lower is better \| \| `latency` \| 64.579 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65535 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.04 pp sourcePopulation 138 stderr 1.043 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/engineering/google~1gemini-3.1-pro-preview` | | | | | | | | | [Public Benefits Bench v1 / multi turn](https://benchize.com/benchmarks/vals-ai%3Apublic-benefits-bench-v1%3Amulti_turn.md) | 52.91% | SE ±1.3 pp | 5 / 8 | $0.09 | 109.31source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_snap_v1.BPa0Hrre.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 52.909 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.087001 \| USD \| per_test \| Lower is better \| \| `latency` \| 109.306 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.3 pp sourcePopulation 13 stderr 1.298 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_snap_v1.BPa0Hrre.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:18.363Z Evidence SHA-256 `b85e2b3cbdf03e13e2ffdb27ce4f108003a548d705a24ae26a0a115e2027475d` Source locator `/tasks/multi_turn/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / bankruptcy restructuring](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Abankruptcy_restructuring.md) | 0% | SE ±0 pp | 3 / 49 | $0.6 | 252.1source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.604154 \| USD \| per_test \| Lower is better \| \| `latency` \| 252.102 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/bankruptcy_restructuring/google~1gemini-3.1-pro-preview` | | | | | | | | | [MMLU Pro / philosophy](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aphilosophy.md) | 90.38% | SE ±1.32 pp | 2 / 40 | $0.02 | 15.57source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 90.381 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.017428 \| USD \| per_test \| Lower is better \| \| `latency` \| 15.572 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65535 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.32 pp sourcePopulation 138 stderr 1.32 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/philosophy/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / environmental esg](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aenvironmental_esg.md) | 0% | SE ±0 pp | 1 / 49 | $0.53 | 223.29source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.526047 \| USD \| per_test \| Lower is better \| \| `latency` \| 223.294 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/environmental_esg/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate corporate ma](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__corporate_ma.md) | 59.62% | SE ±2.78 pp | 43 / 49 | $0.46 | 189.98source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 59.615 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.464081 \| USD \| per_test \| Lower is better \| \| `latency` \| 189.98 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.78 pp sourcePopulation 81 stderr 2.782 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__corporate_ma/google~1gemini-3.1-pro-preview` | | | | | | | | | [MortgageTax / overall](https://benchize.com/benchmarks/vals-ai%3Amortgage_tax%3Aoverall.md) | 69.4% | SE ±0.91 pp | 6 / 31 | $0.02 | 23.62source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mortgage_tax.DCd-oO5Y.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.396 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.023542 \| USD \| per_test \| Lower is better \| \| `latency` \| 23.616 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.91 pp sourcePopulation 98 stderr 0.908 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mortgage_tax.DCd-oO5Y.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:58.500Z Evidence SHA-256 `e5c844561f97d119d66d70269e726ecfb68be79eae39acf14ead3e95422e648c` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [Vals Multimodal Index / finance agent](https://benchize.com/benchmarks/vals-ai%3Avals_multimodal_index%3Afinance_agent.md) | 42.97% | SE ±1.93 pp | 16 / 20 | $1.38 | 196.07source duration | [Vals AI · 1.2](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 42.968 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.378409 \| USD \| per_test \| Lower is better \| \| `latency` \| 196.068 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.93 pp sourcePopulation 33 stderr 1.931 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/finance_agent/google~1gemini-3.1-pro-preview` | | | | | | | | | [ProofBench v1.1 / overall](https://benchize.com/benchmarks/vals-ai%3Aproof_bench%3Aoverall.md) | 26% | SE ±4.41 pp | 28 / 34 | $0.78 | 889.06source duration | [Vals AI · 1.1](https://www.vals.ai/_astro/benchmark_view_proof_bench.i3Hwh8rW.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 26 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.784253 \| USD \| per_test \| Lower is better \| \| `latency` \| 889.059 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.41 pp sourcePopulation 52 stderr 4.408 tokenTotals {"cache_read_tokens":47066256,"cache_write_tokens":null,"input_tokens":68260388,"output_tokens":2100047,"reasoning_tokens":691664} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_proof_bench.i3Hwh8rW.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1.1 Observed 2026-10-21T16:46:01.401Z Evidence SHA-256 `22be2aca25de5c4a9eb6448a95713e09a7c97126dfc6f4f75c4afd0e563209cb` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [MysteryMechanism / overall](https://benchize.com/benchmarks/vals-ai%3Amysterymechanism%3Aoverall.md) | 20.72% | SE ±2.73 pp | 15 / 19 | $1.07 | 360.46source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mysterymechanism.75uyyp4P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 20.721 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.065147 \| USD \| per_test \| Lower is better \| \| `latency` \| 360.456 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.73 pp sourcePopulation 27 stderr 2.726 tokenTotals {"cache_read_tokens":214402702,"cache_write_tokens":0,"input_tokens":262123547,"output_tokens":8178370,"reasoning_tokens":3546375} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mysterymechanism.75uyyp4P.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:46:04.766Z Evidence SHA-256 `1ef09cd0a0ee8690b0c07b2272057461242359b1680e24975ea42971e03acea9` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / litigation dispute resolution](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Alitigation_dispute_resolution.md) | 0% | SE ±0 pp | 6 / 49 | $0.72 | 476.51source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.721268 \| USD \| per_test \| Lower is better \| \| `latency` \| 476.506 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/litigation_dispute_resolution/google~1gemini-3.1-pro-preview` | | | | | | | | | [Legal Research Bench / family](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Afamily.md) | 4.55% | SE ±4.55 pp | 39 / 50 | $4.14 | 434.38source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 4.545 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.138935 \| USD \| per_test \| Lower is better \| \| `latency` \| 434.377 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.55 pp sourcePopulation 82 stderr 4.545 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/family/google~1gemini-3.1-pro-preview` | | | | | | | | | [DeepSWE](https://benchize.com/benchmarks/deepswe%3Av1.1.md) | 11.73% | Range 10.24–13.21% | 68 / 70 | $2.14 | 867.8s | [deepswe](https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `pass_at_1` \| 0.1172566371681416 \| score_fraction \| scored_rollout_attempts \| Higher is better \| \| `pass_at_4` \| 0.2831858407079646 \| score_fraction \| tasks_with_one_passing_rollout \| Higher is better \| \| `cost_per_attempt` \| 2.1434045568888886 \| USD \| scored_attempt \| Lower is better \| \| `duration` \| 867.7979107013274 \| seconds \| scored_attempt \| Lower is better \| ### Configuration harness mini-swe-agent Reasoning effort high sourceConfiguration mini_swe_agent_gemini_3_1_pro_preview_high ### Reported statistics Uncertainty Reported interval: 10.24–13.21% interval [0.10244568255705315,0.13206759177923005] intervalMethod 95% run-to-run: SE across repeated whole-benchmark passes (1.96 * std(runs)/sqrt(R)) runCount 4 sampleSize 452 taskCount 113 ### Source [Original source](https://deepswe.datacurve.ai/artifacts/v1.1/leaderboard-live.json) Source deepswe Original model label gemini-3-1-pro-preview Observed 2026-10-21T16:30:23.371Z Evidence SHA-256 `a7c15d66288fd249c020b9931c017b92d1a3b90e480b3ff34974b752bd030019` Source locator `/rows/67` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / real estate](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Areal_estate.md) | 0% | SE ±0 pp | 6 / 49 | $3.07 | 546.05source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 3.068252 \| USD \| per_test \| Lower is better \| \| `latency` \| 546.054 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/real_estate/google~1gemini-3.1-pro-preview` | | | | | | | | | [Vals Multimodal Index / overall](https://benchize.com/benchmarks/vals-ai%3Avals_multimodal_index%3Aoverall.md) | 56.07% | SE ±1.41 pp | 16 / 20 | $0.98 | 322.91source duration | [Vals AI · 1.2](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 56.068 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.977777 \| USD \| per_test \| Lower is better \| \| `latency` \| 322.907 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.41 pp sourcePopulation 33 stderr 1.406 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | | [Finance Agent (v2) / earnings analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Aearnings_analysis.md) | 64.65% | SE ±0.65 pp | 37 / 50 | $0.51 | 154.84source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.654 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.509894 \| USD \| per_test \| Lower is better \| \| `latency` \| 154.84 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.65 pp sourcePopulation 80 stderr 0.648 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:45:57.035Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/earnings_analysis/google~1gemini-3.1-pro-preview` | | | | | | | | | [MortgageTax / numerical extraction](https://benchize.com/benchmarks/vals-ai%3Amortgage_tax%3Anumerical_extraction.md) | 76.39% | SE ±1.2 pp | 12 / 31 | $0.02 | 24.79source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mortgage_tax.DCd-oO5Y.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 76.391 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.024731 \| USD \| per_test \| Lower is better \| \| `latency` \| 24.789 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.2 pp sourcePopulation 98 stderr 1.197 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mortgage_tax.DCd-oO5Y.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:58.500Z Evidence SHA-256 `e5c844561f97d119d66d70269e726ecfb68be79eae39acf14ead3e95422e648c` Source locator `/tasks/numerical_extraction/google~1gemini-3.1-pro-preview` | | | | | | | | | [Tax Agent Bench / weighted pass rule source lookup](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Aweighted_pass__rule_source_lookup.md) | 28.26% | SE ±8.04 pp | 41 / 46 | $0.25 | 86.67source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 28.261 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.246919 \| USD \| per_test \| Lower is better \| \| `latency` \| 86.667 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.04 pp sourcePopulation 73 stderr 8.04 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/weighted_pass__rule_source_lookup/google~1gemini-3.1-pro-preview` | | | | | | | | [Next page](https://benchize.com/models/gemini-3-1-pro-preview-high.md?cursor=63ae5c1ab076cd0021c1adf204059056a27675504b1f7501002cffbf813011ee) --- Cite as: Gemini 3.1 Pro PreviewHigh — Benchize, https://benchize.com/models/gemini-3-1-pro-preview-high (retrieved 2026-10-21)