# Gemini 3.5 Flash-LiteHigh > Source: Benchize (https://benchize.com) — https://benchize.com/models/gemini-3-5-flash-lite-high. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/models/gemini-3-5-flash-lite-high. Google Canonical URL: https://benchize.com/models/gemini-3-5-flash-lite-high Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Google ## Benchmarks | Benchmark | Score | ± | Rank | Cost | Latency | Source | | | --- | --- | --- | --- | --- | --- | --- | --- | | [MMLU Pro / other](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aother.md) | 83.12% | SE ±1.23 pp | 22 / 40 | $0.00283 | 3.38source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.117 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.002828 \| USD \| per_test \| Lower is better \| \| `latency` \| 3.379 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.23 pp sourcePopulation 138 stderr 1.232 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/other/google~1gemini-3.5-flash-lite` | | | | | | | | | [Legal Research Bench / health](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Ahealth.md) | 13.33% | SE ±6.31 pp | 43 / 50 | $0.13 | 48.2source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 13.333 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.13408 \| USD \| per_test \| Lower is better \| \| `latency` \| 48.195 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.31 pp sourcePopulation 82 stderr 6.312 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/health/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / arbitration international dispute resolution](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aarbitration_international_dispute_resolution.md) | 0% | SE ±0 pp | 4 / 49 | $0.29 | 226.36source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.288514 \| USD \| per_test \| Lower is better \| \| `latency` \| 226.358 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/arbitration_international_dispute_resolution/google~1gemini-3.5-flash-lite` | | | | | | | | | [MMLU Pro / business](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Abusiness.md) | 90.62% | SE ±1.04 pp | 25 / 40 | $0.00445 | 4.54source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 90.621 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.004449 \| USD \| per_test \| Lower is better \| \| `latency` \| 4.543 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.04 pp sourcePopulation 138 stderr 1.038 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/business/google~1gemini-3.5-flash-lite` | | | | | | | | | [MortgageTax / numerical extraction](https://benchize.com/benchmarks/vals-ai%3Amortgage_tax%3Anumerical_extraction.md) | 75.84% | SE ±1.21 pp | 14 / 31 | $0.00537 | 5.86source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mortgage_tax.DCd-oO5Y.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.835 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.00537 \| USD \| per_test \| Lower is better \| \| `latency` \| 5.856 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.21 pp sourcePopulation 98 stderr 1.207 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mortgage_tax.DCd-oO5Y.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:58.500Z Evidence SHA-256 `e5c844561f97d119d66d70269e726ecfb68be79eae39acf14ead3e95422e648c` Source locator `/tasks/numerical_extraction/google~1gemini-3.5-flash-lite` | | | | | | | | | [EMB / dcf](https://benchize.com/benchmarks/vals-ai%3Aemb%3Adcf.md) | 33.61% | SE ±7.56 pp | 41 / 47 | $1.27 | 529.93source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 33.614 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.27137 \| USD \| per_test \| Lower is better \| \| `latency` \| 529.933 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.56 pp sourcePopulation 76 stderr 7.561 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dcf/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate tax](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__tax.md) | 62.88% | SE ±2.8 pp | 38 / 49 | $0.25 | 231.34source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.876 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.251258 \| USD \| per_test \| Lower is better \| \| `latency` \| 231.336 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.8 pp sourcePopulation 81 stderr 2.799 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__tax/google~1gemini-3.5-flash-lite` | | | | | | | | | [ARC-AGI-1](https://benchize.com/benchmarks/arc-prize%3Av1_Semi_Private.md) | 53.5% | — | 135 / 213 | 0.09usd per task | — | [arc prize · v1](https://arcprize.org/media/data/leaderboard/v1.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 0.535 \| score_fraction \| v1_Semi_Private \| Higher is better \| \| `cost_per_task` \| 0.092850542 \| usd_per_task \| v1_Semi_Private \| Lower is better \| ### Configuration evaluationSplit v1_Semi_Private harness Not reported modelGroup gemini-3-5-flash-lite modelType CoT Reasoning effort high sourceModelVersion gemini-3-5-flash-lite-high thinkingEnabled true ### Source [Original source](https://arcprize.org/media/data/leaderboard/v1.json) Source arc prize Original model label Gemini 3.5 Flash-Lite (High) Source version v1 Observed 2026-10-21T16:34:20.378Z Evidence SHA-256 `302d8267c0522bf450ec1deeac4d060e2cf2fb444ba0a4269014d9535de0bd1f` Source locator `/evaluations/200` | | | | | | | | | [Legal Research Bench / overall](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aoverall.md) | 13.94% | SE ±2.41 pp | 43 / 50 | $0.12 | 59.2source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 13.942 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.124723 \| USD \| per_test \| Lower is better \| \| `latency` \| 59.197 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.41 pp sourcePopulation 82 stderr 2.408 tokenTotals {"cache_read_tokens":56268106,"cache_write_tokens":0,"input_tokens":121402055,"output_tokens":1885651,"reasoning_tokens":1270438} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | | [LegalBench / overall](https://benchize.com/benchmarks/vals-ai%3Alegal_bench%3Aoverall.md) | 84.07% | SE ±0.44 pp | 17 / 44 | $0.00128 | 1.67source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.073 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.00128 \| USD \| per_test \| Lower is better \| \| `latency` \| 1.666 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.44 pp sourcePopulation 151 stderr 0.437 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.390Z Evidence SHA-256 `c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | | [MMLU Pro / law](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Alaw.md) | 72.48% | SE ±1.35 pp | 25 / 40 | $0.00537 | 6.53source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.48 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.005367 \| USD \| per_test \| Lower is better \| \| `latency` \| 6.529 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.35 pp sourcePopulation 138 stderr 1.346 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/law/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / structured finance securitization](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Astructured_finance_securitization.md) | 0% | SE ±0 pp | 12 / 49 | $0.3 | 206.17source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.29865 \| USD \| per_test \| Lower is better \| \| `latency` \| 206.166 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/structured_finance_securitization/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate bankruptcy restructuring](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__bankruptcy_restructuring.md) | 61.59% | SE ±2.87 pp | 39 / 49 | $0.34 | 108.95source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 61.592 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.341968 \| USD \| per_test \| Lower is better \| \| `latency` \| 108.948 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.87 pp sourcePopulation 81 stderr 2.866 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__bankruptcy_restructuring/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / tax](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Atax.md) | 0% | SE ±0 pp | 3 / 49 | $0.25 | 231.34source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.251258 \| USD \| per_test \| Lower is better \| \| `latency` \| 231.336 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/tax/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate capital markets](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__capital_markets.md) | 77.65% | SE ±2.61 pp | 37 / 49 | $0.25 | 97.31source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.647 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.251837 \| USD \| per_test \| Lower is better \| \| `latency` \| 97.31 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.61 pp sourcePopulation 81 stderr 2.614 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__capital_markets/google~1gemini-3.5-flash-lite` | | | | | | | | | [Legal Research Bench / weighted pass health](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aweighted_pass__health.md) | 75.74% | SE ±3.9 pp | 42 / 50 | $0.13 | 48.2source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.743 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.13408 \| USD \| per_test \| Lower is better \| \| `latency` \| 48.195 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.9 pp sourcePopulation 82 stderr 3.902 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/weighted_pass__health/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / weighted pass](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Aweighted_pass.md) | 37.45% | SE ±3.12 pp | 40 / 46 | $0.08 | 47.46source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 37.451 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.081536 \| USD \| per_test \| Lower is better \| \| `latency` \| 47.461 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.12 pp sourcePopulation 73 stderr 3.115 tokenTotals {"cache_read_tokens":42212661,"cache_write_tokens":null,"input_tokens":74874616,"output_tokens":1868593,"reasoning_tokens":1185933} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/weighted_pass/google~1gemini-3.5-flash-lite` | | | | | | | | | [MMLU Pro / chemistry](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Achemistry.md) | 90.99% | SE ±0.85 pp | 21 / 40 | $0.00659 | 6.09source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 90.989 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.006593 \| USD \| per_test \| Lower is better \| \| `latency` \| 6.086 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.85 pp sourcePopulation 138 stderr 0.851 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/chemistry/google~1gemini-3.5-flash-lite` | | | | | | | | | [SWE-bench / overall](https://benchize.com/benchmarks/vals-ai%3Aswebench%3Aoverall.md) | 75% | SE ±1.94 pp | 22 / 39 | $0.31 | 439.67source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.30898 \| USD \| per_test \| Lower is better \| \| `latency` \| 439.669 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.94 pp sourcePopulation 88 stderr 1.938 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:13.711Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / rule source lookup](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Arule_source_lookup.md) | 13.33% | SE ±6.31 pp | 40 / 46 | $0.06 | 41.27source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 13.333 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.060332 \| USD \| per_test \| Lower is better \| \| `latency` \| 41.274 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.31 pp sourcePopulation 73 stderr 6.312 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/rule_source_lookup/google~1gemini-3.5-flash-lite` | | | | | | | | | [MMLU Pro / engineering](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aengineering.md) | 82.77% | SE ±1.21 pp | 23 / 40 | $0.00993 | 9.67source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.766 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.009929 \| USD \| per_test \| Lower is better \| \| `latency` \| 9.671 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.21 pp sourcePopulation 138 stderr 1.213 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/engineering/google~1gemini-3.5-flash-lite` | | | | | | | | | [MMLU Pro / health](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Ahealth.md) | 80.44% | SE ±1.39 pp | 24 / 40 | $0.00316 | 3.87source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.44 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.003157 \| USD \| per_test \| Lower is better \| \| `latency` \| 3.872 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.39 pp sourcePopulation 138 stderr 1.387 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/health/google~1gemini-3.5-flash-lite` | | | | | | | | | [Finance Agent (v2) / earnings analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Aearnings_analysis.md) | 64.37% | SE ±1.23 pp | 38 / 50 | $0.23 | 381.43source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.369 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.230543 \| USD \| per_test \| Lower is better \| \| `latency` \| 381.425 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.23 pp sourcePopulation 80 stderr 1.232 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2 Observed 2026-10-21T16:45:57.035Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/earnings_analysis/google~1gemini-3.5-flash-lite` | | | | | | | | | [Legal Research Bench / immigration](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aimmigration.md) | 18.18% | SE ±12.2 pp | 32 / 50 | $0.1 | 60.52source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 18.182 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.102995 \| USD \| per_test \| Lower is better \| \| `latency` \| 60.521 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±12.2 pp sourcePopulation 82 stderr 12.197 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/immigration/google~1gemini-3.5-flash-lite` | | | | | | | | | [ARC-AGI-2](https://benchize.com/benchmarks/arc-prize%3Av2_Semi_Private.md) | 10.28% | — | 119 / 212 | 0.14usd per task | — | [arc prize · v2](https://arcprize.org/media/data/leaderboard/v2.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 0.10277777777777779 \| score_fraction \| v2_Semi_Private \| Higher is better \| \| `cost_per_task` \| 0.13899638250000004 \| usd_per_task \| v2_Semi_Private \| Lower is better \| ### Configuration evaluationSplit v2_Semi_Private harness Not reported modelGroup gemini-3-5-flash-lite modelType CoT Reasoning effort high sourceModelVersion gemini-3-5-flash-lite-high thinkingEnabled true ### Source [Original source](https://arcprize.org/media/data/leaderboard/v2.json) Source arc prize Original model label Gemini 3.5 Flash-Lite (High) Source version v2 Observed 2026-10-21T16:34:20.750Z Evidence SHA-256 `bca68e12cc9a01954ae21b641844c1ecbabe146cb24b87f0f8e9b1db27433c29` Source locator `/evaluations/202` | | | | | | | | | [LiveCodeBench / hard](https://benchize.com/benchmarks/vals-ai%3Alcb%3Ahard.md) | 53.43% | SE ±2.67 pp | 34 / 43 | $0.04 | 39.68source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 53.429 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.035699 \| USD \| per_test \| Lower is better \| \| `latency` \| 39.678 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.67 pp sourcePopulation 143 stderr 2.666 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:14.220Z Evidence SHA-256 `1a7c7dabc9d95a0182886b1edc42623987cac2095a0d407a5e334f37acdfa2f5` Source locator `/tasks/hard/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / intellectual property](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aintellectual_property.md) | 0% | SE ±0 pp | 6 / 49 | $0.4 | 129.78source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.400486 \| USD \| per_test \| Lower is better \| \| `latency` \| 129.784 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/intellectual_property/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate banking finance](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__banking_finance.md) | 79.93% | SE ±2.45 pp | 39 / 49 | $0.35 | 335.92source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 79.926 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.353865 \| USD \| per_test \| Lower is better \| \| `latency` \| 335.918 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.45 pp sourcePopulation 81 stderr 2.447 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__banking_finance/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / banking finance](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Abanking_finance.md) | 0% | SE ±0 pp | 5 / 49 | $0.35 | 335.92source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.353865 \| USD \| per_test \| Lower is better \| \| `latency` \| 335.918 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/banking_finance/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / white collar defense investigations](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Awhite_collar_defense_investigations.md) | 0% | SE ±0 pp | 1 / 49 | $0.26 | 282.53source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.260621 \| USD \| per_test \| Lower is better \| \| `latency` \| 282.53 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/white_collar_defense_investigations/google~1gemini-3.5-flash-lite` | | | | | | | | | [TaxEval v2 / overall](https://benchize.com/benchmarks/vals-ai%3Atax_eval_v2%3Aoverall.md) | 72.61% | SE ±0.88 pp | 26 / 40 | $0.00729 | 8.82source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.608 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.00729 \| USD \| per_test \| Lower is better \| \| `latency` \| 8.815 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.88 pp sourcePopulation 145 stderr 0.875 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2 Observed 2026-10-21T16:45:59.000Z Evidence SHA-256 `8b354a6ae848f5bb25a2e8b061b5f576d5b9e0c11b941dc1e6bed078e6529cd6` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate arbitration international dispute resolution](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__arbitration_international_dispute_resolution.md) | 63.8% | SE ±2.88 pp | 38 / 49 | $0.29 | 226.36source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.799 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.288514 \| USD \| per_test \| Lower is better \| \| `latency` \| 226.358 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.88 pp sourcePopulation 81 stderr 2.882 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__arbitration_international_dispute_resolution/google~1gemini-3.5-flash-lite` | | | | | | | | | [CorpFin v2 / max fitting context](https://benchize.com/benchmarks/vals-ai%3Acorp_fin_v2%3Amax_fitting_context.md) | 62.35% | SE ±1.65 pp | 23 / 39 | $0.06 | 4.54source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.354 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.059711 \| USD \| per_test \| Lower is better \| \| `latency` \| 4.541 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.65 pp sourcePopulation 134 stderr 1.654 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2 Observed 2026-10-21T16:45:57.998Z Evidence SHA-256 `7cf2217ab3728d5a84bffce82aab9fc52d7f810b0cbe8c65b387df47f3d31c9b` Source locator `/tasks/max_fitting_context/google~1gemini-3.5-flash-lite` | | | | | | | | | [CyberBench v1.1 / poc](https://benchize.com/benchmarks/benchmark%3Acyber%3Apoc.md) | 45% | SE ±6.42 pp | 18 / 28 | $0.49 | 1,452.84source duration | [Vals AI · 1.1](https://www.vals.ai/_astro/benchmark_view_cyberbench.DS6UOlu0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 45 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.486741 \| USD \| per_test \| Lower is better \| \| `latency` \| 1452.835 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.42 pp sourcePopulation 47 stderr 6.423 tokenTotals {"cache_read_tokens":324526199,"cache_write_tokens":null,"input_tokens":377107947,"output_tokens":1477655,"reasoning_tokens":1071214} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_cyberbench.DS6UOlu0.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1.1 Observed 2026-10-21T16:46:15.238Z Evidence SHA-256 `943f593f929f8971a6b746269a2851b5febc2aad4e2f1066895f715033cdc7f1` Source locator `/tasks/poc/google~1gemini-3.5-flash-lite` | | | | | | | | | [τ³-Banking](https://benchize.com/benchmarks/mercor%3Atau3-banking.md) | 18.6% | — | 23 / 24 | — | — | [mercor](https://www.mercor.com/apex/partner-benchmarks/tau-3-banking-leaderboard/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `pass_1` \| 18.6 \| percent \| task_completion_pass_at_1 \| Higher is better \| ### Configuration datasetUrl Not reported datasetVersion Not reported harness Not reported Reasoning effort high ### Reported statistics reportedErrors {"pass_1":6} sampleSize 291 ### Source [Original source](https://www.mercor.com/apex/partner-benchmarks/tau-3-banking-leaderboard/) Source mercor Original model label Gemini 3.5 Flash Lite Observed 2026-10-21T16:33:27.812Z Evidence SHA-256 `78850adeb8ad047fcb1dd65cc04ffaca61c86fc67e7035f9973805b02483ed07` Source locator `/__NEXT_DATA__/props/pageProps/leaderboard/rows/17` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate employment labor](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__employment_labor.md) | 65.54% | SE ±2.91 pp | 39 / 49 | $0.34 | 121.51source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 65.543 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.340745 \| USD \| per_test \| Lower is better \| \| `latency` \| 121.512 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.91 pp sourcePopulation 81 stderr 2.914 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__employment_labor/google~1gemini-3.5-flash-lite` | | | | | | | | | [LiveCodeBench / medium](https://benchize.com/benchmarks/vals-ai%3Alcb%3Amedium.md) | 84.86% | SE ±1.83 pp | 32 / 43 | $0.02 | 18.41source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.856 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.01721 \| USD \| per_test \| Lower is better \| \| `latency` \| 18.406 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.83 pp sourcePopulation 143 stderr 1.832 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:14.220Z Evidence SHA-256 `1a7c7dabc9d95a0182886b1edc42623987cac2095a0d407a5e334f37acdfa2f5` Source locator `/tasks/medium/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / funds asset management](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Afunds_asset_management.md) | 0% | SE ±0 pp | 11 / 49 | $0.31 | 309.59source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.313043 \| USD \| per_test \| Lower is better \| \| `latency` \| 309.586 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/funds_asset_management/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate funds asset management](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__funds_asset_management.md) | 62.68% | SE ±2.62 pp | 39 / 49 | $0.31 | 309.59source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.682 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.313043 \| USD \| per_test \| Lower is better \| \| `latency` \| 309.586 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.62 pp sourcePopulation 81 stderr 2.615 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__funds_asset_management/google~1gemini-3.5-flash-lite` | | | | | | | | | [EMB / ma](https://benchize.com/benchmarks/vals-ai%3Aemb%3Ama.md) | 43.21% | SE ±8 pp | 41 / 47 | $0.95 | 417.57source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 43.213 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.952387 \| USD \| per_test \| Lower is better \| \| `latency` \| 417.573 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8 pp sourcePopulation 76 stderr 7.997 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / healthcare life sciences](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Ahealthcare_life_sciences.md) | 0% | SE ±0 pp | 24 / 49 | $0.28 | 141.4source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.284177 \| USD \| per_test \| Lower is better \| \| `latency` \| 141.404 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/healthcare_life_sciences/google~1gemini-3.5-flash-lite` | | | | | | | | | [Finance Agent (v2) / comparables](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Acomparables.md) | 45.19% | SE ±0.6 pp | 27 / 50 | $0.54 | 433.72source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 45.186 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.542906 \| USD \| per_test \| Lower is better \| \| `latency` \| 433.722 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.6 pp sourcePopulation 80 stderr 0.602 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2 Observed 2026-10-21T16:45:57.035Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/comparables/google~1gemini-3.5-flash-lite` | | | | | | | | | [Legal Research Bench / administrative regulatory](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aadministrative_regulatory.md) | 19.7% | SE ±4.93 pp | 41 / 50 | $0.12 | 50.78source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 19.697 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.123387 \| USD \| per_test \| Lower is better \| \| `latency` \| 50.778 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.93 pp sourcePopulation 82 stderr 4.933 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/administrative_regulatory/google~1gemini-3.5-flash-lite` | | | | | | | | | [AutomationBench](https://benchize.com/benchmarks/zapier%3Aautomationbench.md) | 3.35% | — | 95 / 113 | $0.13 | — | [zapier](https://framerusercontent.com/sites/4WTSl4BNjd1q9QFEFibC6h/lOvLnhItj8S8-1vyjH_s33Jn-WmUXPmiJ5Diajl0Q38.CMutaHYl.mjs) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 3.35 \| percent \| reported \| Higher is better \| \| `cost` \| 0.13 \| USD \| per_task \| Lower is better \| ### Configuration benchmarkVersion 1.0.6 costFootnote false Fallback policy Not reported Reasoning effort high Model configuration Gemini 3.5 Flash Lite (High) thinkingEnabled true ### Reported statistics rank 108 ### Source [Original source](https://framerusercontent.com/sites/4WTSl4BNjd1q9QFEFibC6h/lOvLnhItj8S8-1vyjH_s33Jn-WmUXPmiJ5Diajl0Q38.CMutaHYl.mjs) Source zapier Original model label Gemini 3.5 Flash Lite (High) Observed 2026-10-21T16:34:56.783Z Evidence SHA-256 `43978e38f018789fa59c11de5328abe9e5905bfa9a5a4d601fa3aa4d9c153b0d` Source locator `/rows/107` | | | | | | | | | [APEX-Agents](https://benchize.com/benchmarks/mercor%3Aapex-agents%3A1.1.md) | 29.3% | — | 35 / 38 | — | — | [mercor · 1.1](https://www.mercor.com/apex/apex-agents-leaderboard/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `pass_1` \| 29.3 \| percent \| task_completion_pass_at_1 \| Higher is better \| \| `mean_score` \| 46.1 \| percent \| rubric_mean \| Higher is better \| ### Configuration datasetUrl https://huggingface.co/datasets/mercor/apex-agents-v1.1 datasetVersion 1.1 harness loop_truncated_tools_agent Reasoning effort high ### Reported statistics reportedErrors {"mean_score":4.7,"pass_1":4.9} sampleSize 954 ### Source [Original source](https://www.mercor.com/apex/apex-agents-leaderboard/) Source mercor Original model label Gemini 3.5 Flash Lite Source version 1.1 Observed 2026-10-21T16:33:27.509Z Evidence SHA-256 `359c569c3b0b62ab375895eabf3c45846fa45551d419d5c9d6a4c9117db78184` Source locator `/__NEXT_DATA__/props/pageProps/leaderboardData/45` | | | | | | | | | [LegalBench / rule tasks](https://benchize.com/benchmarks/vals-ai%3Alegal_bench%3Arule_tasks.md) | 81.71% | SE ±1.6 pp | 17 / 44 | $0.00162 | 2.34source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 81.708 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.001616 \| USD \| per_test \| Lower is better \| \| `latency` \| 2.344 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.6 pp sourcePopulation 151 stderr 1.602 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.390Z Evidence SHA-256 `c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52` Source locator `/tasks/rule_tasks/google~1gemini-3.5-flash-lite` | | | | | | | | | [MMLU Pro / psychology](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Apsychology.md) | 85.59% | SE ±1.24 pp | 24 / 40 | $0.00257 | 3.2source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.589 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.002565 \| USD \| per_test \| Lower is better \| \| `latency` \| 3.199 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.24 pp sourcePopulation 138 stderr 1.243 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/psychology/google~1gemini-3.5-flash-lite` | | | | | | | | | [EMB / scratch](https://benchize.com/benchmarks/vals-ai%3Aemb%3Ascratch.md) | 44.93% | SE ±2.94 pp | 41 / 47 | $1.15 | 477.72source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 44.928 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.149494 \| USD \| per_test \| Lower is better \| \| `latency` \| 477.721 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.94 pp sourcePopulation 76 stderr 2.935 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / numbers calculations](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Anumbers_calculations.md) | 20% | SE ±7.43 pp | 38 / 46 | $0.06 | 44.32source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 20 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.059752 \| USD \| per_test \| Lower is better \| \| `latency` \| 44.318 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.43 pp sourcePopulation 73 stderr 7.428 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/numbers_calculations/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / corporate governance](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acorporate_governance.md) | 0% | SE ±0 pp | 11 / 49 | $0.39 | 139.3source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.392709 \| USD \| per_test \| Lower is better \| \| `latency` \| 139.304 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/corporate_governance/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate data privacy cybersecurity](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__data_privacy_cybersecurity.md) | 72.44% | SE ±2.66 pp | 39 / 49 | $0.32 | 298.16source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.438 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.320555 \| USD \| per_test \| Lower is better \| \| `latency` \| 298.162 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.66 pp sourcePopulation 81 stderr 2.661 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__data_privacy_cybersecurity/google~1gemini-3.5-flash-lite` | | | | | | | | | [Vibe Code Bench v1.1 / overall](https://benchize.com/benchmarks/vals-ai%3Avibe-code%3Aoverall.md) | 37.16% | SE ±4.63 pp | 44 / 59 | $0.83 | 501.52source duration | [Vals AI · 1.1](https://www.vals.ai/_astro/benchmark_view_vibe_code_bench.DcEBMbCA.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 37.161 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.833111 \| USD \| per_test \| Lower is better \| \| `latency` \| 501.522 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness OpenHands maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.63 pp sourcePopulation 116 stderr 4.627 tokenTotals {"cache_read_tokens":307681641,"cache_write_tokens":0,"input_tokens":387765124,"output_tokens":3360015,"reasoning_tokens":658865} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vibe_code_bench.DcEBMbCA.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1.1 Observed 2026-10-21T16:46:08.625Z Evidence SHA-256 `0e8c23f521a0f751c5592952620e197b376cade41c78578461b7aa5740a173e7` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | | [ProgramBench / strict](https://benchize.com/benchmarks/vals-ai%3Aprogrambench%3Astrict.md) | 0% | SE ±0 pp | 23 / 40 | $0.34 | 435source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.337485 \| USD \| per_test \| Lower is better \| \| `latency` \| 434.999 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 64 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:10.137Z Evidence SHA-256 `e9bbb4378c7a058441a21d456baf9316b96c2cf5b58d371a3c7c803f5000bd8c` Source locator `/tasks/strict/google~1gemini-3.5-flash-lite` | | | | | | | | | [Legal Research Bench / weighted pass constitutional civil rights](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aweighted_pass__constitutional_civil_rights.md) | 77.48% | SE ±3.78 pp | 41 / 50 | $0.08 | 61.41source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.484 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.080105 \| USD \| per_test \| Lower is better \| \| `latency` \| 61.408 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.78 pp sourcePopulation 82 stderr 3.778 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/weighted_pass__constitutional_civil_rights/google~1gemini-3.5-flash-lite` | | | | | | | | | [MMLU Pro / philosophy](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aphilosophy.md) | 82.97% | SE ±1.68 pp | 23 / 40 | $0.00344 | 4.24source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.966 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.003441 \| USD \| per_test \| Lower is better \| \| `latency` \| 4.243 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.68 pp sourcePopulation 138 stderr 1.683 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/philosophy/google~1gemini-3.5-flash-lite` | | | | | | | | | [SWE-bench / <15 min fix](https://benchize.com/benchmarks/vals-ai%3Aswebench%3A%3C15%20min%20fix.md) | 85.57% | SE ±2.53 pp | 25 / 39 | $0.23 | 325.53source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.567 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.229026 \| USD \| per_test \| Lower is better \| \| `latency` \| 325.532 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.53 pp sourcePopulation 88 stderr 2.53 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:13.711Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/<15 min fix/google~1gemini-3.5-flash-lite` | | | | | | | | | [Chess Puzzles](https://benchize.com/benchmarks/epoch-ai%3Achess_puzzles.md) | 16% | SE ±3.68 pp | 51 / 63 | — | — | [epoch ai · Chess Puzzles](https://epoch.ai/data/benchmark_data.zip#file=chess_puzzles.csv) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 0.16 \| score_fraction \| Best score (across scorers) \| Higher is better \| \| `mean_score` \| 0.16 \| score_fraction \| mean_score \| Higher is better \| ### Configuration benchmarkVersion Chess Puzzles reasoningBudget Not reported Reasoning effort high sourceModelVersion gemini-3.5-flash-lite_high ### Reported statistics Uncertainty Standard error: ±3.68 pp standardError 0.036845294917747066 ### Source [Original source](https://epoch.ai/data/benchmark_data.zip#file=chess_puzzles.csv) Source epoch ai Original model label gemini-3.5-flash-lite Source version Chess Puzzles Evaluated 2026-10-13T13:26:52.000Z Observed 2026-10-21T16:30:24.199Z Evidence SHA-256 `fa0a449c370b0cceaf273d9472078cce3e5aef6be7009bddf115e6df7f930c54` Source locator `/rows/119` | | | | | | | | | [Terminal-Bench 2.1 / easy](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Aeasy.md) | 66.67% | SE ±8.33 pp | 37 / 43 | $7.5 | 9,264.53source duration | [Vals AI · 2.1](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.667 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 7.499411 \| USD \| per_test \| Lower is better \| \| `latency` \| 9264.533 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2.1 Observed 2026-10-21T16:46:13.216Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/google~1gemini-3.5-flash-lite` | | | | | | | | | [EMB / misc](https://benchize.com/benchmarks/vals-ai%3Aemb%3Amisc.md) | 48.82% | SE ±6.17 pp | 37 / 47 | $0.68 | 307.15source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 48.822 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.676879 \| USD \| per_test \| Lower is better \| \| `latency` \| 307.154 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.17 pp sourcePopulation 76 stderr 6.172 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/misc/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / energy natural resources](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aenergy_natural_resources.md) | 0% | SE ±0 pp | 28 / 49 | $0.33 | 360.4source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.330094 \| USD \| per_test \| Lower is better \| \| `latency` \| 360.398 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/energy_natural_resources/google~1gemini-3.5-flash-lite` | | | | | | | | | [CyberBench v1.1 / patch](https://benchize.com/benchmarks/benchmark%3Acyber%3Apatch.md) | 73.21% | SE ±5.92 pp | 27 / 28 | $0.36 | 698.94source duration | [Vals AI · 1.1](https://www.vals.ai/_astro/benchmark_view_cyberbench.DS6UOlu0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.214 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.363507 \| USD \| per_test \| Lower is better \| \| `latency` \| 698.939 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.92 pp sourcePopulation 47 stderr 5.918 tokenTotals {"cache_read_tokens":183828171,"cache_write_tokens":null,"input_tokens":222465635,"output_tokens":1300133,"reasoning_tokens":1087005} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_cyberbench.DS6UOlu0.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1.1 Observed 2026-10-21T16:46:15.238Z Evidence SHA-256 `943f593f929f8971a6b746269a2851b5febc2aad4e2f1066895f715033cdc7f1` Source locator `/tasks/patch/google~1gemini-3.5-flash-lite` | | | | | | | | | [Legal Research Bench / family](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Afamily.md) | 4.55% | SE ±4.55 pp | 39 / 50 | $0.17 | 62.26source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 4.545 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.169797 \| USD \| per_test \| Lower is better \| \| `latency` \| 62.256 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.55 pp sourcePopulation 82 stderr 4.545 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/family/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / weighted pass forms filings](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Aweighted_pass__forms_filings.md) | 31.66% | SE ±7.79 pp | 38 / 46 | $0.11 | 75.87source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 31.655 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.112503 \| USD \| per_test \| Lower is better \| \| `latency` \| 75.867 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.79 pp sourcePopulation 73 stderr 7.794 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/weighted_pass__forms_filings/google~1gemini-3.5-flash-lite` | | | | | | | | | [FrontierMath Tier 4 v2 (private)](https://benchize.com/benchmarks/epoch-ai%3Afrontiermath_tier_4_v2.md) | 0% | SE ±0 pp | 37 / 38 | — | — | [epoch ai · FrontierMath-Tier-4-v2-Private](https://epoch.ai/data/benchmark_data.zip#file=frontiermath_tier_4_v2.csv) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 0.0 \| score_fraction \| Best score (across scorers) \| Higher is better \| \| `mean_score` \| 0.0 \| score_fraction \| mean_score \| Higher is better \| ### Configuration benchmarkVersion FrontierMath-Tier-4-v2-Private reasoningBudget Not reported Reasoning effort high sourceModelVersion gemini-3.5-flash-lite_high ### Reported statistics Uncertainty Standard error: ±0 pp standardError 0 ### Source [Original source](https://epoch.ai/data/benchmark_data.zip#file=frontiermath_tier_4_v2.csv) Source epoch ai Original model label gemini-3.5-flash-lite Source version FrontierMath-Tier-4-v2-Private Evaluated 2026-08-02T01:20:17.000Z Observed 2026-10-21T16:30:24.199Z Evidence SHA-256 `48b886c828092231989fba6e3819efb7462e89ef681b7b306b4bfa3505829e4a` Source locator `/rows/33` | | | | | | | | | [Legal Research Bench / weighted pass](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aweighted_pass.md) | 68.94% | SE ±1.85 pp | 42 / 50 | $0.12 | 59.2source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.939 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.124723 \| USD \| per_test \| Lower is better \| \| `latency` \| 59.197 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.85 pp sourcePopulation 82 stderr 1.846 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/weighted_pass/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / antitrust competition](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aantitrust_competition.md) | 0% | SE ±0 pp | 7 / 49 | $0.46 | 186.1source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.456896 \| USD \| per_test \| Lower is better \| \| `latency` \| 186.102 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/antitrust_competition/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / fact pattern analysis](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Afact_pattern_analysis.md) | 4.65% | SE ±3.25 pp | 42 / 46 | $0.09 | 60.34source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 4.651 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.086864 \| USD \| per_test \| Lower is better \| \| `latency` \| 60.341 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.25 pp sourcePopulation 73 stderr 3.249 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/fact_pattern_analysis/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / employment labor](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aemployment_labor.md) | 0% | SE ±0 pp | 1 / 49 | $0.34 | 121.51source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.340745 \| USD \| per_test \| Lower is better \| \| `latency` \| 121.512 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/employment_labor/google~1gemini-3.5-flash-lite` | | | | | | | | | [SAGE / overall](https://benchize.com/benchmarks/vals-ai%3Asage%3Aoverall.md) | 47.3% | SE ±3.38 pp | 22 / 41 | $0.01 | 10.75source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 47.303 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.010542 \| USD \| per_test \| Lower is better \| \| `latency` \| 10.751 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 30000 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.38 pp sourcePopulation 91 stderr 3.384 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_sage.RRwIIzO5.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:07.515Z Evidence SHA-256 `cc5fb3fecdb0382d492ff167a42ad84b9c5c3883373c78ed87baaa972b8b6f7c` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / immigration](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aimmigration.md) | 0% | SE ±0 pp | 6 / 49 | $0.28 | 431.31source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.284081 \| USD \| per_test \| Lower is better \| \| `latency` \| 431.31 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/immigration/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / weighted pass fact pattern analysis](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Aweighted_pass__fact_pattern_analysis.md) | 32.25% | SE ±6.33 pp | 40 / 46 | $0.09 | 60.34source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 32.253 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.086864 \| USD \| per_test \| Lower is better \| \| `latency` \| 60.341 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.33 pp sourcePopulation 73 stderr 6.329 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/weighted_pass__fact_pattern_analysis/google~1gemini-3.5-flash-lite` | | | | | | | | | [LegalBench / interpretation tasks](https://benchize.com/benchmarks/vals-ai%3Alegal_bench%3Ainterpretation_tasks.md) | 80.33% | SE ±0.34 pp | 25 / 44 | $0.00118 | 1.49source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.332 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.001183 \| USD \| per_test \| Lower is better \| \| `latency` \| 1.492 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.34 pp sourcePopulation 151 stderr 0.336 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.390Z Evidence SHA-256 `c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52` Source locator `/tasks/interpretation_tasks/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate international trade sanctions](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__international_trade_sanctions.md) | 73.81% | SE ±2.78 pp | 42 / 49 | $0.33 | 271.01source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.81 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.333493 \| USD \| per_test \| Lower is better \| \| `latency` \| 271.008 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.78 pp sourcePopulation 81 stderr 2.775 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__international_trade_sanctions/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / forms filings](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Aforms_filings.md) | 0% | SE ±0 pp | 41 / 46 | $0.11 | 75.87source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.112503 \| USD \| per_test \| Lower is better \| \| `latency` \| 75.867 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 73 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/forms_filings/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / weighted pass current temporal analysis](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Aweighted_pass__current_temporal_analysis.md) | 24.92% | SE ±7.27 pp | 43 / 46 | $0.08 | 52.71source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 24.916 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.075425 \| USD \| per_test \| Lower is better \| \| `latency` \| 52.71 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.27 pp sourcePopulation 73 stderr 7.269 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/weighted_pass__current_temporal_analysis/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / real estate](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Areal_estate.md) | 0% | SE ±0 pp | 6 / 49 | $0.44 | 397.49source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.444015 \| USD \| per_test \| Lower is better \| \| `latency` \| 397.494 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/real_estate/google~1gemini-3.5-flash-lite` | | | | | | | | | [Code Migration / overall](https://benchize.com/benchmarks/vals-ai%3Acode-migration%3Aoverall.md) | 6.15% | SE ±1.71 pp | 43 / 47 | $0.53 | 1,322.3source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 6.148 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.525186 \| USD \| per_test \| Lower is better \| \| `latency` \| 1322.302 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.71 pp sourcePopulation 79 stderr 1.708 tokenTotals {"cache_read_tokens":277179852,"cache_write_tokens":null,"input_tokens":422878800,"output_tokens":4819392,"reasoning_tokens":2204805} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:12.712Z Evidence SHA-256 `99194597f2c76c402195be47839409046e18bd30454616da6dd40b736a088fa4` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | | [EMB / comps](https://benchize.com/benchmarks/vals-ai%3Aemb%3Acomps.md) | 46.42% | SE ±7.78 pp | 36 / 47 | $1.78 | 591.25source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 46.417 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.779222 \| USD \| per_test \| Lower is better \| \| `latency` \| 591.245 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.78 pp sourcePopulation 76 stderr 7.777 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:57.495Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/google~1gemini-3.5-flash-lite` | | | | | | | | | [LegalBench / rhetoric tasks](https://benchize.com/benchmarks/vals-ai%3Alegal_bench%3Arhetoric_tasks.md) | 80.6% | SE ±1.11 pp | 22 / 44 | $0.00111 | 1.38source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.595 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.001105 \| USD \| per_test \| Lower is better \| \| `latency` \| 1.379 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.11 pp sourcePopulation 151 stderr 1.109 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.390Z Evidence SHA-256 `c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52` Source locator `/tasks/rhetoric_tasks/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / overall](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Aoverall.md) | 7.25% | SE ±1.87 pp | 42 / 46 | $0.08 | 47.46source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 7.254 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.081536 \| USD \| per_test \| Lower is better \| \| `latency` \| 47.461 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.87 pp sourcePopulation 73 stderr 1.872 tokenTotals {"cache_read_tokens":42212661,"cache_write_tokens":null,"input_tokens":74874616,"output_tokens":1868593,"reasoning_tokens":1185933} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate trusts estates private client](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__trusts_estates_private_client.md) | 75.57% | SE ±2.66 pp | 34 / 49 | $0.23 | 104.81source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.573 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.232718 \| USD \| per_test \| Lower is better \| \| `latency` \| 104.812 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.66 pp sourcePopulation 81 stderr 2.66 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__trusts_estates_private_client/google~1gemini-3.5-flash-lite` | | | | | | | | | [Legal Research Bench / weighted pass criminal](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aweighted_pass__criminal.md) | 68.11% | SE ±3.36 pp | 42 / 50 | $0.12 | 63.5source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.108 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.122864 \| USD \| per_test \| Lower is better \| \| `latency` \| 63.499 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.36 pp sourcePopulation 82 stderr 3.364 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/weighted_pass__criminal/google~1gemini-3.5-flash-lite` | | | | | | | | | [Finance Agent (v2) / market analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Amarket_analysis.md) | 57.02% | SE ±1.96 pp | 35 / 50 | $0.25 | 266.66source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.022 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.245829 \| USD \| per_test \| Lower is better \| \| `latency` \| 266.658 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.96 pp sourcePopulation 80 stderr 1.964 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2 Observed 2026-10-21T16:45:57.035Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/market_analysis/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate litigation dispute resolution](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__litigation_dispute_resolution.md) | 74.91% | SE ±2.66 pp | 36 / 49 | $0.36 | 104.6source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 74.906 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.356731 \| USD \| per_test \| Lower is better \| \| `latency` \| 104.598 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.66 pp sourcePopulation 81 stderr 2.658 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__litigation_dispute_resolution/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate antitrust competition](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__antitrust_competition.md) | 76.45% | SE ±2.56 pp | 38 / 49 | $0.46 | 186.1source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 76.449 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.456896 \| USD \| per_test \| Lower is better \| \| `latency` \| 186.102 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.56 pp sourcePopulation 81 stderr 2.559 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__antitrust_competition/google~1gemini-3.5-flash-lite` | | | | | | | | | [Finance Agent (v2) / general qualitative analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Ageneral_qualitative_analysis.md) | 51.11% | SE ±0.24 pp | 39 / 50 | $0.28 | 150.92source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 51.11 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.275972 \| USD \| per_test \| Lower is better \| \| `latency` \| 150.92 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.24 pp sourcePopulation 80 stderr 0.239 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2 Observed 2026-10-21T16:45:57.035Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/general_qualitative_analysis/google~1gemini-3.5-flash-lite` | | | | | | | | | [Legal Research Bench / criminal](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Acriminal.md) | 12.28% | SE ±4.39 pp | 42 / 50 | $0.12 | 63.5source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 12.281 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.122864 \| USD \| per_test \| Lower is better \| \| `latency` \| 63.499 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.39 pp sourcePopulation 82 stderr 4.386 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:52.389Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/criminal/google~1gemini-3.5-flash-lite` | | | | | | | | | [Finance Agent (v2) / general quantitative analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Ageneral_quantitative_analysis.md) | 66.12% | SE ±2.59 pp | 37 / 50 | $0.43 | 291.9source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.12 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.427377 \| USD \| per_test \| Lower is better \| \| `latency` \| 291.898 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.59 pp sourcePopulation 80 stderr 2.586 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2 Observed 2026-10-21T16:45:57.035Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/general_quantitative_analysis/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate structured finance securitization](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__structured_finance_securitization.md) | 64.44% | SE ±2.7 pp | 40 / 49 | $0.3 | 206.17source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.444 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.29865 \| USD \| per_test \| Lower is better \| \| `latency` \| 206.166 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.7 pp sourcePopulation 81 stderr 2.701 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__structured_finance_securitization/google~1gemini-3.5-flash-lite` | | | | | | | | | [LegalBench / conclusion tasks](https://benchize.com/benchmarks/vals-ai%3Alegal_bench%3Aconclusion_tasks.md) | 88.4% | SE ±0.72 pp | 25 / 44 | $0.00131 | 1.78source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 88.404 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.001313 \| USD \| per_test \| Lower is better \| \| `latency` \| 1.784 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.72 pp sourcePopulation 151 stderr 0.721 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.390Z Evidence SHA-256 `c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52` Source locator `/tasks/conclusion_tasks/google~1gemini-3.5-flash-lite` | | | | | | | | | [TaxEval v2 / stepwise reasoning](https://benchize.com/benchmarks/vals-ai%3Atax_eval_v2%3Astepwise_reasoning.md) | 83.48% | SE ±1.06 pp | 26 / 40 | $0.00729 | 8.82source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 83.483 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.00729 \| USD \| per_test \| Lower is better \| \| `latency` \| 8.815 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.06 pp sourcePopulation 145 stderr 1.062 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2 Observed 2026-10-21T16:45:59.000Z Evidence SHA-256 `8b354a6ae848f5bb25a2e8b061b5f576d5b9e0c11b941dc1e6bed078e6529cd6` Source locator `/tasks/stepwise_reasoning/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate real estate](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__real_estate.md) | 64.14% | SE ±2.59 pp | 36 / 49 | $0.44 | 397.49source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.14 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.444015 \| USD \| per_test \| Lower is better \| \| `latency` \| 397.494 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.59 pp sourcePopulation 81 stderr 2.593 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__real_estate/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / emerging companies venture capital](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aemerging_companies_venture_capital.md) | 0% | SE ±0 pp | 18 / 49 | $0.36 | 193.59source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.358329 \| USD \| per_test \| Lower is better \| \| `latency` \| 193.59 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/emerging_companies_venture_capital/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate corporate ma](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__corporate_ma.md) | 77.24% | SE ±2.38 pp | 40 / 49 | $0.33 | 248.88source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.244 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.330445 \| USD \| per_test \| Lower is better \| \| `latency` \| 248.88 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.38 pp sourcePopulation 81 stderr 2.377 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__corporate_ma/google~1gemini-3.5-flash-lite` | | | | | | | | | [ProgramBench / partial](https://benchize.com/benchmarks/vals-ai%3Aprogrambench%3Apartial.md) | 33.85% | SE ±1.76 pp | 33 / 40 | $0.34 | 435source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 33.852 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.337485 \| USD \| per_test \| Lower is better \| \| `latency` \| 434.999 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.76 pp sourcePopulation 64 stderr 1.758 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:10.137Z Evidence SHA-256 `e9bbb4378c7a058441a21d456baf9316b96c2cf5b58d371a3c7c803f5000bd8c` Source locator `/tasks/partial/google~1gemini-3.5-flash-lite` | | | | | | | | | [Tax Agent Bench / controversy precedence analysis](https://benchize.com/benchmarks/vals-ai%3Atax_agent_bench%3Acontroversy_precedence_analysis.md) | 3.33% | SE ±3.33 pp | 32 / 46 | $0.09 | 59.71source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 3.333 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.092032 \| USD \| per_test \| Lower is better \| \| `latency` \| 59.71 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.33 pp sourcePopulation 73 stderr 3.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_agent_bench.BFpexnpS.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:55.958Z Evidence SHA-256 `446db182b67efb7923b3187a4040ec3a3ea492b8ad699a2b6d708d714cc063ac` Source locator `/tasks/controversy_precedence_analysis/google~1gemini-3.5-flash-lite` | | | | | | | | | [Finance Agent (v2) / precedents](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Aprecedents.md) | 25.97% | SE ±0.77 pp | 26 / 50 | $0.58 | 540.07source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 25.972 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.58475 \| USD \| per_test \| Lower is better \| \| `latency` \| 540.069 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.77 pp sourcePopulation 80 stderr 0.772 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 2 Observed 2026-10-21T16:45:57.035Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/precedents/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / data privacy cybersecurity](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Adata_privacy_cybersecurity.md) | 0% | SE ±0 pp | 13 / 49 | $0.32 | 298.16source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.320555 \| USD \| per_test \| Lower is better \| \| `latency` \| 298.162 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/data_privacy_cybersecurity/google~1gemini-3.5-flash-lite` | | | | | | | | | [Harvey's Legal Agent Benchmark v1 / overall](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aoverall.md) | 0% | SE ±0 pp | 42 / 49 | $0.33 | 232.9source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.331805 \| USD \| per_test \| Lower is better \| \| `latency` \| 232.895 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals {"cache_read_tokens":146158528,"cache_write_tokens":null,"input_tokens":234750879,"output_tokens":3541635,"reasoning_tokens":1058834} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:45:53.909Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/overall/google~1gemini-3.5-flash-lite` | | | | | | | | | [MMLU Pro / math](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Amath.md) | 94.45% | SE ±0.62 pp | 22 / 40 | $0.00529 | 4.91source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 94.449 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.005287 \| USD \| per_test \| Lower is better \| \| `latency` \| 4.909 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.62 pp sourcePopulation 138 stderr 0.623 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label google/gemini-3.5-flash-lite Source version 1 Observed 2026-10-21T16:46:06.503Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/math/google~1gemini-3.5-flash-lite` | | | | | | | | [Next page](https://benchize.com/models/gemini-3-5-flash-lite-high.md?cursor=9440348d8a56f1693633ef887075b3867c5955cbf552d38ad1f5b5f13c7882a8) --- Cite as: Gemini 3.5 Flash-LiteHigh — Benchize, https://benchize.com/models/gemini-3-5-flash-lite-high (retrieved 2026-10-21)