# CaseLaw v2 / overall > Source: Benchize (https://benchize.com) — https://benchize.com/benchmarks/vals-ai%3Acase_law_v2%3Aoverall. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/benchmarks/vals-ai%3Acase_law_v2%3Aoverall. Private question-answer benchmark over Canadian court-cases. Canonical URL: https://benchize.com/benchmarks/vals-ai%3Acase_law_v2%3Aoverall Last updated: Wed, 21 Oct 2026 18:52:20 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Private question-answer benchmark over Canadian court-cases. Professional work · Models tested: 17 · Score publishers: Vals AI ## Model results | # | Model | Score | ± | Cost | Latency | | | --- | --- | --- | --- | --- | --- | --- | | 1 | [Claude Opus 4.7](https://benchize.com/models/claude-opus-4-7.md) | 68.38% | SE ±0.43 pp | $0.16 | 17.68source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.381 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.161535 \| USD \| per_test \| Lower is better \| \| `latency` \| 17.677 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort unknown Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.43 pp sourcePopulation 54 stderr 0.429 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label anthropic/claude-opus-4-7 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/anthropic~1claude-opus-4-7` | | | | | | | | 2 | [GPT-5.5](https://benchize.com/models/gpt-5-5-xhigh.md)Extra high | 66.24% | SE ±0.32 pp | $0.16 | 25.2source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 66.238 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.155011 \| USD \| per_test \| Lower is better \| \| `latency` \| 25.2 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider OpenAI reasoning Not reported Reasoning effort xhigh Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.32 pp sourcePopulation 54 stderr 0.322 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label openai/gpt-5.5 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/openai~1gpt-5.5` | | | | | | | | 3 | [Gemini 3.1 Pro Preview](https://benchize.com/models/gemini-3-1-pro-preview-high.md)High | 64.85% | SE ±1.24 pp | $0.14 | 56.79source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.845 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.136185 \| USD \| per_test \| Lower is better \| \| `latency` \| 56.785 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.24 pp sourcePopulation 54 stderr 1.236 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/google~1gemini-3.1-pro-preview` | | | | | | | | 4 | [Command A](https://benchize.com/models/command-a.md) | 64.52% | SE ±1.09 pp | $0.1 | 41.56source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.523 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.098244 \| USD \| per_test \| Lower is better \| \| `latency` \| 41.555 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 8192 provider Cohere reasoning Not reported Reasoning effort unknown Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.09 pp sourcePopulation 54 stderr 1.088 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label cohere/command-a-03-2025 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/cohere~1command-a-03-2025` | | | | | | | | 5 | [Claude Sonnet 4.6](https://benchize.com/models/claude-sonnet-4-6.md) | 63.99% | SE ±0.56 pp | $0.14 | 36.2source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.987 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.144942 \| USD \| per_test \| Lower is better \| \| `latency` \| 36.196 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Anthropic reasoning Not reported Reasoning effort unknown Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.56 pp sourcePopulation 54 stderr 0.557 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label anthropic/claude-sonnet-4-6 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/anthropic~1claude-sonnet-4-6` | | | | | | | | 6 | [Gemini 2.5 Pro](https://benchize.com/models/gemini-2-5-pro.md) | 63.88% | SE ±0.6 pp | $0.26 | 44.02source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 63.88 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.257186 \| USD \| per_test \| Lower is better \| \| `latency` \| 44.02 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort unknown Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.6 pp sourcePopulation 54 stderr 0.597 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label google/gemini-2.5-pro Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/google~1gemini-2.5-pro` | | | | | | | | 7 | [Mistral Large 3 2512](https://benchize.com/models/mistral-large-2512.md) | 61.42% | SE ±0.97 pp | $0.02 | 40.84source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 61.415 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.018968 \| USD \| per_test \| Lower is better \| \| `latency` \| 40.839 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 256000 provider Mistral AI reasoning Not reported Reasoning effort unknown Temperature 0.065 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.97 pp sourcePopulation 54 stderr 0.965 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label mistralai/mistral-large-2512 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/mistralai~1mistral-large-2512` | | | | | | | | 8 | [MiniMax M2.7](https://benchize.com/models/minimax-m2-7.md) | 60.88% | SE ±0.77 pp | $0.00309 | 21.64source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 60.879 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.003091 \| USD \| per_test \| Lower is better \| \| `latency` \| 21.639 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 196608 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.77 pp sourcePopulation 54 stderr 0.773 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label minimax/MiniMax-M2.7 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/minimax~1MiniMax-M2.7` | | | | | | | | 9 | [GPT-4o (2024-11-20)](https://benchize.com/models/gpt-4o-2024-11-20-high.md)High | 59.7% | SE ±0.43 pp | $0.07 | 40.87source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 59.7 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.066774 \| USD \| per_test \| Lower is better \| \| `latency` \| 40.867 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 16384 provider OpenAI reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.43 pp sourcePopulation 54 stderr 0.429 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label openai/gpt-4o-2024-11-20 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/openai~1gpt-4o-2024-11-20` | | | | | | | | 10 | [DeepSeek V4 Pro](https://benchize.com/models/deepseek-v4-pro-max.md)Max | 59.38% | SE ±1.5 pp | $0.01 | 48.44source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 59.378 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.013152 \| USD \| per_test \| Lower is better \| \| `latency` \| 48.444 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider DeepSeek reasoning Not reported Reasoning effort max Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.5 pp sourcePopulation 54 stderr 1.5 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label deepseek/deepseek-v4-pro Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/deepseek~1deepseek-v4-pro` | | | | | | | | 11 | [Trinity Large Thinking](https://benchize.com/models/trinity-large-thinking.md) | 57.88% | SE ±1.86 pp | $0.00887 | 14.55source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.878 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.008867 \| USD \| per_test \| Lower is better \| \| `latency` \| 14.554 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider Arcee-Ai reasoning Not reported Reasoning effort unknown Temperature 0.3 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.86 pp sourcePopulation 54 stderr 1.856 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label arcee-ai/trinity-large-thinking Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/arcee-ai~1trinity-large-thinking` | | | | | | | | 12 | [MiniMax M2.1](https://benchize.com/models/minimax-m2-1.md) | 55.84% | SE ±0.47 pp | $0.00801 | 37.36source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 55.842 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.008008 \| USD \| per_test \| Lower is better \| \| `latency` \| 37.357 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 131000 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.47 pp sourcePopulation 54 stderr 0.467 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label minimax/MiniMax-M2.1 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/minimax~1MiniMax-M2.1` | | | | | | | | 12 | [Gemini 3 Flash Preview](https://benchize.com/models/gemini-3-flash-preview-high.md)High | 55.84% | SE ±0.11 pp | $0.03 | 260.5source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 55.842 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.025399 \| USD \| per_test \| Lower is better \| \| `latency` \| 260.502 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.11 pp sourcePopulation 54 stderr 0.107 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label google/gemini-3-flash-preview Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/google~1gemini-3-flash-preview` | | | | | | | | 14 | [Gemini 3.1 Flash Lite Preview](https://benchize.com/models/gemini-3-1-flash-lite-preview-high.md)High | 54.98% | SE ±0.74 pp | $0.00931 | 4.64source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 54.984 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.009312 \| USD \| per_test \| Lower is better \| \| `latency` \| 4.641 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 65536 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.74 pp sourcePopulation 54 stderr 0.742 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label google/gemini-3.1-flash-lite-preview Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/google~1gemini-3.1-flash-lite-preview` | | | | | | | | 15 | [MiniMax M2.5](https://benchize.com/models/minimax-m2-5.md) | 53.48% | SE ±0.47 pp | $0.01 | 30.88source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 53.483 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.012109 \| USD \| per_test \| Lower is better \| \| `latency` \| 30.875 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 196608 provider MiniMax reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.47 pp sourcePopulation 54 stderr 0.467 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label minimax/MiniMax-M2.5 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/minimax~1MiniMax-M2.5` | | | | | | | | 16 | [Gemma 4 31B](https://benchize.com/models/gemma-4-31b-it-high.md)High | 52.63% | SE ±1.34 pp | $0 | 28.72source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 52.626 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0 \| USD \| per_test \| Lower is better \| \| `latency` \| 28.716 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 32768 provider Google reasoning Not reported Reasoning effort high Temperature 1 topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.34 pp sourcePopulation 54 stderr 1.343 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label google/gemma-4-31b-it Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/google~1gemma-4-31b-it` | | | | | | | | 17 | [Mistral Medium 3.5](https://benchize.com/models/mistral-medium-3-5-high.md)High | 44.16% | SE ±1.06 pp | $0.08 | 64.64source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 44.159 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.084987 \| USD \| per_test \| Lower is better \| \| `latency` \| 64.635 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 80000 provider Mistral AI reasoning Not reported Reasoning effort high Temperature 0.7 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.06 pp sourcePopulation 54 stderr 1.056 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_case_law_v2.DLKA06_7.json) Source Vals AI Original model label mistralai/mistral-medium-3.5 Source version 2 Observed 2026-10-21T16:45:54.497Z Evidence SHA-256 `293f388f7e0fe563287d4cd19beaa7ef9821b32795dace34bd0197ed47a592d4` Source locator `/tasks/overall/mistralai~1mistral-medium-3.5` | | | | | | | --- Cite as: CaseLaw v2 / overall — Benchize, https://benchize.com/benchmarks/vals-ai%3Acase_law_v2%3Aoverall (retrieved 2026-10-21)