# Nemotron 3 Ultra > Source: Benchize (https://benchize.com) — https://benchize.com/models/nemotron-3-ultra-550b-a55b. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/models/nemotron-3-ultra-550b-a55b. NVIDIA Canonical URL: https://benchize.com/models/nemotron-3-ultra-550b-a55b Last updated: Thu, 22 Oct 2026 04:40:22 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. NVIDIA ## Benchmarks | Benchmark | Score | ± | Rank | Latency | Source | | | --- | --- | --- | --- | --- | --- | --- | | [Harvey's Legal Agent Benchmark v1 / trusts estates private client](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Atrusts_estates_private_client.md) | 0% | SE ±0 pp | 15 / 49 | 1,141.15source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 1141.15 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/trusts_estates_private_client/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate immigration](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__immigration.md) | 84.36% | SE ±2.19 pp | 26 / 49 | 3,031.8source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.364 \| percent \| reported \| Higher is better \| \| `latency` \| 3031.804 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.19 pp sourcePopulation 81 stderr 2.194 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__immigration/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / corporate ma](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acorporate_ma.md) | 0% | SE ±0 pp | 21 / 49 | 2,821.81source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 2821.808 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/corporate_ma/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate corporate governance](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__corporate_governance.md) | 72.95% | SE ±2.6 pp | 40 / 49 | 1,552.81source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.945 \| percent \| reported \| Higher is better \| \| `latency` \| 1552.808 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.6 pp sourcePopulation 81 stderr 2.604 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__corporate_governance/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate healthcare life sciences](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__healthcare_life_sciences.md) | 69.64% | SE ±2.65 pp | 39 / 49 | 1,443.01source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.637 \| percent \| reported \| Higher is better \| \| `latency` \| 1443.008 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.65 pp sourcePopulation 81 stderr 2.646 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__healthcare_life_sciences/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Terminal-Bench 2.1 / medium](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Amedium.md) | 62.42% | SE ±2.42 pp | 34 / 43 | 945.21source duration | [Vals AI · 2.1](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 62.424 \| percent \| reported \| Higher is better \| \| `latency` \| 945.205 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.42 pp sourcePopulation 76 stderr 2.424 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2.1 Observed 2026-10-22T02:10:27.811Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/medium/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate insurance](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__insurance.md) | 82.78% | SE ±2.29 pp | 30 / 49 | 3,435.6source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.784 \| percent \| reported \| Higher is better \| \| `latency` \| 3435.596 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.29 pp sourcePopulation 81 stderr 2.289 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__insurance/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / capital markets](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acapital_markets.md) | 0% | SE ±0 pp | 16 / 49 | 975.16source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 975.162 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/capital_markets/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / insurance](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Ainsurance.md) | 0% | SE ±0 pp | 2 / 49 | 3,435.6source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 3435.596 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/insurance/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [TaxEval v2 / stepwise reasoning](https://benchize.com/benchmarks/vals-ai%3Atax_eval_v2%3Astepwise_reasoning.md) | 84.55% | SE ±1.03 pp | 19 / 40 | 36.49source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.546 \| percent \| reported \| Higher is better \| \| `latency` \| 36.485 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.03 pp sourcePopulation 145 stderr 1.034 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:14.414Z Evidence SHA-256 `8b354a6ae848f5bb25a2e8b061b5f576d5b9e0c11b941dc1e6bed078e6529cd6` Source locator `/tasks/stepwise_reasoning/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / philosophy](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aphilosophy.md) | 80.96% | SE ±1.76 pp | 26 / 40 | 16.11source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.962 \| percent \| reported \| Higher is better \| \| `latency` \| 16.114 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.76 pp sourcePopulation 138 stderr 1.758 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/philosophy/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / data privacy cybersecurity](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Adata_privacy_cybersecurity.md) | 0% | SE ±0 pp | 13 / 49 | 2,722.32source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 2722.318 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/data_privacy_cybersecurity/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [EMB / dcf](https://benchize.com/benchmarks/vals-ai%3Aemb%3Adcf.md) | 29.55% | SE ±6.77 pp | 42 / 47 | 2,934.04source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 29.545 \| percent \| reported \| Higher is better \| \| `latency` \| 2934.041 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.77 pp sourcePopulation 76 stderr 6.769 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:12.907Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dcf/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate data privacy cybersecurity](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__data_privacy_cybersecurity.md) | 70.13% | SE ±3.02 pp | 40 / 49 | 2,722.32source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.13 \| percent \| reported \| Higher is better \| \| `latency` \| 2722.318 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.02 pp sourcePopulation 81 stderr 3.018 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__data_privacy_cybersecurity/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / psychology](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Apsychology.md) | 84.59% | SE ±1.28 pp | 27 / 40 | 13.75source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.586 \| percent \| reported \| Higher is better \| \| `latency` \| 13.753 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.28 pp sourcePopulation 138 stderr 1.278 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/psychology/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate funds asset management](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__funds_asset_management.md) | 51.7% | SE ±3.08 pp | 44 / 49 | 3,933.64source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 51.698 \| percent \| reported \| Higher is better \| \| `latency` \| 3933.64 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.08 pp sourcePopulation 81 stderr 3.076 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__funds_asset_management/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate employment labor](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__employment_labor.md) | 77.15% | SE ±2.57 pp | 29 / 49 | 1,238.39source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.154 \| percent \| reported \| Higher is better \| \| `latency` \| 1238.388 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.57 pp sourcePopulation 81 stderr 2.574 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__employment_labor/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Legal Research Bench / immigration](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aimmigration.md) | 9.09% | SE ±9.09 pp | 41 / 50 | 848.64source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 9.091 \| percent \| reported \| Higher is better \| \| `latency` \| 848.642 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.09 pp sourcePopulation 82 stderr 9.091 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:07.934Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/immigration/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Legal Research Bench / family](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Afamily.md) | 4.55% | SE ±4.55 pp | 39 / 50 | 458.97source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 4.545 \| percent \| reported \| Higher is better \| \| `latency` \| 458.965 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.55 pp sourcePopulation 82 stderr 4.545 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:07.934Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/family/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / overall](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aoverall.md) | 0.42% | SE ±0 pp | 40 / 49 | 2,048.54source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0.417 \| percent \| reported \| Higher is better \| \| `latency` \| 2048.539 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals {"cache_read_tokens":48583680,"cache_write_tokens":null,"input_tokens":305706389,"output_tokens":3758581,"reasoning_tokens":null} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / tax](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Atax.md) | 0% | SE ±0 pp | 3 / 49 | 2,605.47source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 2605.465 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/tax/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Legal Research Bench / overall](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aoverall.md) | 15.39% | SE ±2.51 pp | 42 / 50 | 611.37source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 15.385 \| percent \| reported \| Higher is better \| \| `latency` \| 611.366 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.51 pp sourcePopulation 82 stderr 2.508 tokenTotals {"cache_read_tokens":49852416,"cache_write_tokens":0,"input_tokens":181276135,"output_tokens":3143670,"reasoning_tokens":0} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:07.934Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / physics](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aphysics.md) | 92.23% | SE ±0.74 pp | 19 / 40 | 46.02source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 92.225 \| percent \| reported \| Higher is better \| \| `latency` \| 46.02 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.74 pp sourcePopulation 138 stderr 0.743 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/physics/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [SWE-bench / >4 hours](https://benchize.com/benchmarks/vals-ai%3Aswebench%3A%3E4%20hours.md) | 33.33% | SE ±33.33 pp | 13 / 39 | 475.33source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 33.333 \| percent \| reported \| Higher is better \| \| `latency` \| 475.334 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±33.33 pp sourcePopulation 88 stderr 33.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:28.317Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/>4 hours/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Finance Agent (v2) / general quantitative analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Ageneral_quantitative_analysis.md) | 55.8% | SE ±3 pp | 42 / 50 | 213.53source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 55.802 \| percent \| reported \| Higher is better \| \| `latency` \| 213.531 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3 pp sourcePopulation 80 stderr 3.002 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:12.404Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/general_quantitative_analysis/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / engineering](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aengineering.md) | 82.77% | SE ±1.21 pp | 23 / 40 | 132.63source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.766 \| percent \| reported \| Higher is better \| \| `latency` \| 132.631 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.21 pp sourcePopulation 138 stderr 1.213 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/engineering/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [EMB / comps](https://benchize.com/benchmarks/vals-ai%3Aemb%3Acomps.md) | 32.45% | SE ±7.63 pp | 42 / 47 | 1,598.87source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 32.452 \| percent \| reported \| Higher is better \| \| `latency` \| 1598.874 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.63 pp sourcePopulation 76 stderr 7.631 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:12.907Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/comps/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / intellectual property](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aintellectual_property.md) | 0% | SE ±0 pp | 6 / 49 | 1,545.29source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 1545.29 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/intellectual_property/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Finance Agent (v2) / financial modeling](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Afinancial_modeling.md) | 7.07% | SE ±1.77 pp | 42 / 50 | 312.22source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 7.071 \| percent \| reported \| Higher is better \| \| `latency` \| 312.219 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.77 pp sourcePopulation 80 stderr 1.766 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:12.404Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/financial_modeling/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [LegalBench / overall](https://benchize.com/benchmarks/vals-ai%3Alegal_bench%3Aoverall.md) | 82.07% | SE ±0.46 pp | 29 / 44 | 5.18source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.066 \| percent \| reported \| Higher is better \| \| `latency` \| 5.179 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.46 pp sourcePopulation 151 stderr 0.46 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:08.909Z Evidence SHA-256 `c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Legal Research Bench / administrative regulatory](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aadministrative_regulatory.md) | 16.67% | SE ±4.62 pp | 43 / 50 | 476.59source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 16.667 \| percent \| reported \| Higher is better \| \| `latency` \| 476.589 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.62 pp sourcePopulation 82 stderr 4.623 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:07.934Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/administrative_regulatory/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / economics](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aeconomics.md) | 88.51% | SE ±1.1 pp | 25 / 40 | 16.62source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 88.507 \| percent \| reported \| Higher is better \| \| `latency` \| 16.623 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.1 pp sourcePopulation 138 stderr 1.098 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/economics/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Finance Agent (v2) / adjustments](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Aadjustments.md) | 34.45% | SE ±3.05 pp | 41 / 50 | 255.22source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 34.449 \| percent \| reported \| Higher is better \| \| `latency` \| 255.224 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.05 pp sourcePopulation 80 stderr 3.045 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:12.404Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/adjustments/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [LiveCodeBench / medium](https://benchize.com/benchmarks/vals-ai%3Alcb%3Amedium.md) | 90.34% | SE ±1.51 pp | 9 / 43 | 100.61source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 90.339 \| percent \| reported \| Higher is better \| \| `latency` \| 100.608 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.51 pp sourcePopulation 143 stderr 1.51 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:28.824Z Evidence SHA-256 `1a7c7dabc9d95a0182886b1edc42623987cac2095a0d407a5e334f37acdfa2f5` Source locator `/tasks/medium/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Finance Agent (v2) / market analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Amarket_analysis.md) | 41.27% | SE ±1.62 pp | 43 / 50 | 208.88source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 41.27 \| percent \| reported \| Higher is better \| \| `latency` \| 208.881 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.62 pp sourcePopulation 80 stderr 1.624 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:12.404Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/market_analysis/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [ProgramBench / overall](https://benchize.com/benchmarks/vals-ai%3Aprogrambench%3Aoverall.md) | 0% | SE ±0 pp | 23 / 40 | 3,718.56source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 3718.555 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 64 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:24.817Z Evidence SHA-256 `e9bbb4378c7a058441a21d456baf9316b96c2cf5b58d371a3c7c803f5000bd8c` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [EMB / ma](https://benchize.com/benchmarks/vals-ai%3Aemb%3Ama.md) | 37.06% | SE ±7.57 pp | 42 / 47 | 2,678.05source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 37.061 \| percent \| reported \| Higher is better \| \| `latency` \| 2678.053 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.57 pp sourcePopulation 76 stderr 7.567 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:12.907Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/ma/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / chemistry](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Achemistry.md) | 91.7% | SE ±0.82 pp | 17 / 40 | 48.43source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.696 \| percent \| reported \| Higher is better \| \| `latency` \| 48.434 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.82 pp sourcePopulation 138 stderr 0.82 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/chemistry/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / white collar defense investigations](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Awhite_collar_defense_investigations.md) | 0% | SE ±0 pp | 1 / 49 | 2,326source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 2326 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/white_collar_defense_investigations/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate white collar defense investigations](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__white_collar_defense_investigations.md) | 67.34% | SE ±2.73 pp | 37 / 49 | 2,326source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 67.34 \| percent \| reported \| Higher is better \| \| `latency` \| 2326 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.73 pp sourcePopulation 81 stderr 2.726 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__white_collar_defense_investigations/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [GPQA Diamond](https://benchize.com/benchmarks/gpqa-diamond.md) | 86.11% | SE ±1.74 pp | 23 / 40 | 158.47source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_gpqa.ghZZBpyU.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 86.112 \| percent \| reported \| Higher is better \| \| `latency` \| 158.471 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.74 pp sourcePopulation 138 stderr 1.738 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_gpqa.ghZZBpyU.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:20.661Z Evidence SHA-256 `951eb5224097733a9d09d06c14908ae2980e2e640e539f2c284877c5092d7077` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Terminal-Bench 2.1 / easy](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Aeasy.md) | 91.67% | SE ±8.33 pp | 15 / 43 | 12,996.57source duration | [Vals AI · 2.1](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.667 \| percent \| reported \| Higher is better \| \| `latency` \| 12996.574 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.33 pp sourcePopulation 76 stderr 8.333 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2.1 Observed 2026-10-22T02:10:27.811Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/easy/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Finance Agent (v2) / all pass](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Aall_pass.md) | 24.8% | SE ±0.73 pp | 42 / 50 | 253.88source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 24.802 \| percent \| reported \| Higher is better \| \| `latency` \| 253.882 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.73 pp sourcePopulation 80 stderr 0.728 tokenTotals {"cache_read_tokens":0,"cache_write_tokens":null,"input_tokens":408960062,"output_tokens":7705379,"reasoning_tokens":0} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:12.404Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/all_pass/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / biology](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Abiology.md) | 92.19% | SE ±1 pp | 27 / 40 | 14.61source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 92.19 \| percent \| reported \| Higher is better \| \| `latency` \| 14.605 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1 pp sourcePopulation 138 stderr 1.002 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/biology/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [LiveCodeBench / easy](https://benchize.com/benchmarks/vals-ai%3Alcb%3Aeasy.md) | 98.45% | SE ±0.69 pp | 26 / 43 | 16.96source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 98.447 \| percent \| reported \| Higher is better \| \| `latency` \| 16.96 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.69 pp sourcePopulation 143 stderr 0.689 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:28.824Z Evidence SHA-256 `1a7c7dabc9d95a0182886b1edc42623987cac2095a0d407a5e334f37acdfa2f5` Source locator `/tasks/easy/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Finance Agent (v2) / earnings analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Aearnings_analysis.md) | 40.62% | SE ±1.89 pp | 44 / 50 | 203.78source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 40.623 \| percent \| reported \| Higher is better \| \| `latency` \| 203.776 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.89 pp sourcePopulation 80 stderr 1.887 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:12.404Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/earnings_analysis/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [EMB / operating](https://benchize.com/benchmarks/vals-ai%3Aemb%3Aoperating.md) | 28.12% | SE ±6.91 pp | 42 / 47 | 2,684.88source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 28.124 \| percent \| reported \| Higher is better \| \| `latency` \| 2684.882 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.91 pp sourcePopulation 76 stderr 6.907 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:12.907Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/operating/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Legal Research Bench / civil litigation](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Acivil_litigation.md) | 22.22% | SE ±8.15 pp | 36 / 50 | 690.61source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 22.222 \| percent \| reported \| Higher is better \| \| `latency` \| 690.614 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±8.15 pp sourcePopulation 82 stderr 8.153 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:07.934Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/civil_litigation/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / business](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Abusiness.md) | 91% | SE ±1.02 pp | 21 / 40 | 30source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 91.001 \| percent \| reported \| Higher is better \| \| `latency` \| 29.998 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.02 pp sourcePopulation 138 stderr 1.019 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/business/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate arbitration international dispute resolution](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__arbitration_international_dispute_resolution.md) | 51.61% | SE ±3 pp | 42 / 49 | 1,361.62source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 51.613 \| percent \| reported \| Higher is better \| \| `latency` \| 1361.618 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3 pp sourcePopulation 81 stderr 2.997 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__arbitration_international_dispute_resolution/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Finance Agent (v2) / overall](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Aoverall.md) | 37.67% | SE ±0.15 pp | 42 / 50 | 253.88source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 37.667 \| percent \| reported \| Higher is better \| \| `latency` \| 253.882 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.15 pp sourcePopulation 80 stderr 0.149 tokenTotals {"cache_read_tokens":0,"cache_write_tokens":null,"input_tokens":408960062,"output_tokens":7705379,"reasoning_tokens":0} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:12.404Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [EMB / template](https://benchize.com/benchmarks/vals-ai%3Aemb%3Atemplate.md) | 18.65% | SE ±3.83 pp | 44 / 47 | 1,984.68source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 18.647 \| percent \| reported \| Higher is better \| \| `latency` \| 1984.68 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.83 pp sourcePopulation 76 stderr 3.826 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:12.907Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/template/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / other](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Aother.md) | 82.9% | SE ±1.24 pp | 24 / 40 | 15.65source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.9 \| percent \| reported \| Higher is better \| \| `latency` \| 15.648 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.24 pp sourcePopulation 138 stderr 1.239 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/other/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / environmental esg](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aenvironmental_esg.md) | 0% | SE ±0 pp | 1 / 49 | 2,788.35source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 2788.346 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/environmental_esg/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Legal Research Bench / weighted pass family](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aweighted_pass__family.md) | 51.56% | SE ±6.47 pp | 43 / 50 | 458.97source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 51.562 \| percent \| reported \| Higher is better \| \| `latency` \| 458.965 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.47 pp sourcePopulation 82 stderr 6.465 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:07.934Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/weighted_pass__family/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Legal Research Bench / health](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Ahealth.md) | 16.67% | SE ±6.92 pp | 42 / 50 | 538.65source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 16.667 \| percent \| reported \| Higher is better \| \| `latency` \| 538.653 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±6.92 pp sourcePopulation 82 stderr 6.92 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:07.934Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/health/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [EMB / misc](https://benchize.com/benchmarks/vals-ai%3Aemb%3Amisc.md) | 33.6% | SE ±7.38 pp | 42 / 47 | 1,205.3source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 33.598 \| percent \| reported \| Higher is better \| \| `latency` \| 1205.295 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.38 pp sourcePopulation 76 stderr 7.38 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:12.907Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/misc/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate tax](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__tax.md) | 56.23% | SE ±3.15 pp | 42 / 49 | 2,605.47source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 56.225 \| percent \| reported \| Higher is better \| \| `latency` \| 2605.465 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.15 pp sourcePopulation 81 stderr 3.15 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__tax/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Terminal-Bench 2.1 / overall](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Aoverall.md) | 50.94% | SE ±2.93 pp | 34 / 43 | 584.12source duration | [Vals AI · 2.1](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 50.936 \| percent \| reported \| Higher is better \| \| `latency` \| 584.116 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.93 pp sourcePopulation 76 stderr 2.925 tokenTotals {"cache_read_tokens":0,"cache_write_tokens":0,"input_tokens":111289804,"output_tokens":2704899,"reasoning_tokens":0} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2.1 Observed 2026-10-22T02:10:27.811Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [EMB / dataroom](https://benchize.com/benchmarks/vals-ai%3Aemb%3Adataroom.md) | 38.06% | SE ±9.38 pp | 42 / 47 | 1,261.47source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 38.062 \| percent \| reported \| Higher is better \| \| `latency` \| 1261.468 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.38 pp sourcePopulation 76 stderr 9.384 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:12.907Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/dataroom/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [LiveCodeBench / overall](https://benchize.com/benchmarks/vals-ai%3Alcb%3Aoverall.md) | 85.98% | SE ±0.99 pp | 15 / 43 | 161.2source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.976 \| percent \| reported \| Higher is better \| \| `latency` \| 161.2 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.99 pp sourcePopulation 143 stderr 0.992 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_lcb.miyRUvVa.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:28.824Z Evidence SHA-256 `1a7c7dabc9d95a0182886b1edc42623987cac2095a0d407a5e334f37acdfa2f5` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Terminal-Bench 2.1 / hard](https://benchize.com/benchmarks/vals-ai%3Aterminal-bench-2-1%3Ahard.md) | 24.44% | SE ±4.01 pp | 36 / 43 | 1,732.88source duration | [Vals AI · 2.1](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 24.444 \| percent \| reported \| Higher is better \| \| `latency` \| 1732.877 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.01 pp sourcePopulation 76 stderr 4.006 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_terminal_bench_2_1.CdyoObsA.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2.1 Observed 2026-10-22T02:10:27.811Z Evidence SHA-256 `a5306805cdf86495bdcceff6b2c8534a3768b9853f5077895dab975b809b054f` Source locator `/tasks/hard/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Legal Research Bench / weighted pass immigration](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aweighted_pass__immigration.md) | 73.48% | SE ±9.16 pp | 41 / 50 | 848.64source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.479 \| percent \| reported \| Higher is better \| \| `latency` \| 848.642 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±9.16 pp sourcePopulation 82 stderr 9.162 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:07.934Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/weighted_pass__immigration/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / computer science](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Acomputer_science.md) | 89.76% | SE ±1.5 pp | 19 / 40 | 31.77source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 89.756 \| percent \| reported \| Higher is better \| \| `latency` \| 31.773 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.5 pp sourcePopulation 138 stderr 1.498 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/computer_science/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / litigation dispute resolution](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Alitigation_dispute_resolution.md) | 0% | SE ±0 pp | 6 / 49 | 946.89source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 946.888 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/litigation_dispute_resolution/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [SWE-bench / 1-4 hours](https://benchize.com/benchmarks/vals-ai%3Aswebench%3A1-4%20hours.md) | 38.1% | SE ±7.58 pp | 25 / 39 | 400.8source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 38.095 \| percent \| reported \| Higher is better \| \| `latency` \| 400.802 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±7.58 pp sourcePopulation 88 stderr 7.584 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:28.317Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/1-4 hours/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / arbitration international dispute resolution](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aarbitration_international_dispute_resolution.md) | 0% | SE ±0 pp | 4 / 49 | 1,361.62source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 1361.618 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/arbitration_international_dispute_resolution/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / math](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Amath.md) | 95.56% | SE ±0.56 pp | 15 / 40 | 30.91source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 95.559 \| percent \| reported \| Higher is better \| \| `latency` \| 30.914 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.56 pp sourcePopulation 138 stderr 0.56 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/math/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Finance Agent (v2) / general qualitative analysis](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Ageneral_qualitative_analysis.md) | 58.49% | SE ±3.97 pp | 35 / 50 | 231.14source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 58.487 \| percent \| reported \| Higher is better \| \| `latency` \| 231.138 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.97 pp sourcePopulation 80 stderr 3.971 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:12.404Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/general_qualitative_analysis/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / antitrust competition](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aantitrust_competition.md) | 0% | SE ±0 pp | 7 / 49 | 2,430.91source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 2430.912 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/antitrust_competition/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / funds asset management](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Afunds_asset_management.md) | 0% | SE ±0 pp | 11 / 49 | 3,933.64source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 3933.64 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/funds_asset_management/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [SWE-bench / <15 min fix](https://benchize.com/benchmarks/vals-ai%3Aswebench%3A%3C15%20min%20fix.md) | 80.93% | SE ±2.83 pp | 30 / 39 | 214.15source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 80.928 \| percent \| reported \| Higher is better \| \| `latency` \| 214.147 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.83 pp sourcePopulation 88 stderr 2.828 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:28.317Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/<15 min fix/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / law](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Alaw.md) | 72.93% | SE ±1.34 pp | 24 / 40 | 33.37source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.934 \| percent \| reported \| Higher is better \| \| `latency` \| 33.366 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.34 pp sourcePopulation 138 stderr 1.339 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/law/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / bankruptcy restructuring](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Abankruptcy_restructuring.md) | 0% | SE ±0 pp | 3 / 49 | 881.33source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 881.327 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/bankruptcy_restructuring/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate emerging companies venture capital](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__emerging_companies_venture_capital.md) | 70.31% | SE ±2.67 pp | 37 / 49 | 2,558.91source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.307 \| percent \| reported \| Higher is better \| \| `latency` \| 2558.912 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.67 pp sourcePopulation 81 stderr 2.674 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__emerging_companies_venture_capital/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MMLU Pro / health](https://benchize.com/benchmarks/vals-ai%3Ammlu_pro%3Ahealth.md) | 79.46% | SE ±1.41 pp | 27 / 40 | 17.42source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 79.462 \| percent \| reported \| Higher is better \| \| `latency` \| 17.419 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.41 pp sourcePopulation 138 stderr 1.412 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_mmlu_pro.y416pXDi.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:21.194Z Evidence SHA-256 `dab025507f46c716dd8d145e90ec0a737d9c56219853f41e4b1cc3c5e8c222b4` Source locator `/tasks/health/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Code Migration / cobol](https://benchize.com/benchmarks/vals-ai%3Acode-migration%3Acobol.md) | 5.71% | SE ±5.71 pp | 42 / 47 | 3,065.57source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 5.714 \| percent \| reported \| Higher is better \| \| `latency` \| 3065.567 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±5.71 pp sourcePopulation 79 stderr 5.714 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:27.307Z Evidence SHA-256 `99194597f2c76c402195be47839409046e18bd30454616da6dd40b736a088fa4` Source locator `/tasks/cobol/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate bankruptcy restructuring](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__bankruptcy_restructuring.md) | 41.08% | SE ±3.63 pp | 44 / 49 | 881.33source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 41.081 \| percent \| reported \| Higher is better \| \| `latency` \| 881.327 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.63 pp sourcePopulation 81 stderr 3.627 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__bankruptcy_restructuring/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [EMB / scratch](https://benchize.com/benchmarks/vals-ai%3Aemb%3Ascratch.md) | 44.31% | SE ±2.87 pp | 42 / 47 | 2,788.15source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 44.305 \| percent \| reported \| Higher is better \| \| `latency` \| 2788.147 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.87 pp sourcePopulation 76 stderr 2.869 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_emb.BY1LXsAw.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:12.907Z Evidence SHA-256 `d807aee620ea47152a1fca0257a8dd12a1be4512c76cc1744c1ae9e285180918` Source locator `/tasks/scratch/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate litigation dispute resolution](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__litigation_dispute_resolution.md) | 47.19% | SE ±3.06 pp | 45 / 49 | 946.89source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 47.191 \| percent \| reported \| Higher is better \| \| `latency` \| 946.888 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.06 pp sourcePopulation 81 stderr 3.061 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__litigation_dispute_resolution/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / healthcare life sciences](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Ahealthcare_life_sciences.md) | 0% | SE ±0 pp | 24 / 49 | 1,443.01source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 1443.008 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/healthcare_life_sciences/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Finance Agent (v2) / precedents](https://benchize.com/benchmarks/vals-ai%3Afabv2%3Aprecedents.md) | 12.82% | SE ±3.42 pp | 44 / 50 | 333.09source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 12.816 \| percent \| reported \| Higher is better \| \| `latency` \| 333.089 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.42 pp sourcePopulation 80 stderr 3.421 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_finance_agent_v2.DypCOnwd.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:12.404Z Evidence SHA-256 `f731b20a26c126725d125519f2b5841aa45254656dba859a9570d7ce7e5d8191` Source locator `/tasks/precedents/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [TaxEval v2 / overall](https://benchize.com/benchmarks/vals-ai%3Atax_eval_v2%3Aoverall.md) | 73.1% | SE ±0.87 pp | 21 / 40 | 36.49source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.099 \| percent \| reported \| Higher is better \| \| `latency` \| 36.485 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.87 pp sourcePopulation 145 stderr 0.866 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_tax_eval_v2.B3yoffr8.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:14.414Z Evidence SHA-256 `8b354a6ae848f5bb25a2e8b061b5f576d5b9e0c11b941dc1e6bed078e6529cd6` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Code Migration / cli](https://benchize.com/benchmarks/vals-ai%3Acode-migration%3Acli.md) | 4.64% | SE ±0.93 pp | 47 / 47 | 6,272.16source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 4.636 \| percent \| reported \| Higher is better \| \| `latency` \| 6272.162 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.93 pp sourcePopulation 79 stderr 0.929 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_code_migration.Dttzoqk3.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:27.307Z Evidence SHA-256 `99194597f2c76c402195be47839409046e18bd30454616da6dd40b736a088fa4` Source locator `/tasks/cli/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [LegalBench / rhetoric tasks](https://benchize.com/benchmarks/vals-ai%3Alegal_bench%3Arhetoric_tasks.md) | 77.89% | SE ±1.09 pp | 35 / 44 | 3.37source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.885 \| percent \| reported \| Higher is better \| \| `latency` \| 3.366 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.09 pp sourcePopulation 151 stderr 1.088 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_bench.D6uTgA3b.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:08.909Z Evidence SHA-256 `c8486ba756baa5cd3955d035edff67440e8db23e69d62ad1a9e18d15b8c4bc52` Source locator `/tasks/rhetoric_tasks/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [SWE-bench / overall](https://benchize.com/benchmarks/vals-ai%3Aswebench%3Aoverall.md) | 69% | SE ±2.07 pp | 31 / 39 | 283.09source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69 \| percent \| reported \| Higher is better \| \| `latency` \| 283.089 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.07 pp sourcePopulation 88 stderr 2.07 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_swebench.Ae8ni7h0.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:28.317Z Evidence SHA-256 `7a4e2e8485283caac500127419f68ec4bd5414c837ade114fc6afa5b2d34ccaa` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [ProgramBench / partial](https://benchize.com/benchmarks/vals-ai%3Aprogrambench%3Apartial.md) | 9.97% | SE ±1.38 pp | 40 / 40 | 3,718.56source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 9.967 \| percent \| reported \| Higher is better \| \| `latency` \| 3718.555 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.38 pp sourcePopulation 64 stderr 1.376 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_programbench.C_J7ZU4X.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:24.817Z Evidence SHA-256 `e9bbb4378c7a058441a21d456baf9316b96c2cf5b58d371a3c7c803f5000bd8c` Source locator `/tasks/partial/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Vibe Code Bench v1.1 / overall](https://benchize.com/benchmarks/vals-ai%3Avibe-code%3Aoverall.md) | 7.64% | SE ±2.78 pp | 53 / 59 | 2,217.46source duration | [Vals AI · 1.1](https://www.vals.ai/_astro/benchmark_view_vibe_code_bench.DcEBMbCA.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 7.636 \| percent \| reported \| Higher is better \| \| `latency` \| 2217.462 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness OpenHands maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.78 pp sourcePopulation 116 stderr 2.783 tokenTotals {"cache_read_tokens":null,"cache_write_tokens":null,"input_tokens":1965727676,"output_tokens":5440855,"reasoning_tokens":null} ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vibe_code_bench.DcEBMbCA.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1.1 Observed 2026-10-22T02:10:23.531Z Evidence SHA-256 `0e8c23f521a0f751c5592952620e197b376cade41c78578461b7aa5740a173e7` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate real estate](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__real_estate.md) | 53.64% | SE ±2.7 pp | 42 / 49 | 1,700.7source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 53.644 \| percent \| reported \| Higher is better \| \| `latency` \| 1700.696 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.7 pp sourcePopulation 81 stderr 2.696 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__real_estate/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / structured finance securitization](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Astructured_finance_securitization.md) | 0% | SE ±0 pp | 12 / 49 | 2,100.3source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 2100.302 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/structured_finance_securitization/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [MedCode / overall](https://benchize.com/benchmarks/vals-ai%3Amedcode%3Aoverall.md) | 38.62% | SE ±2 pp | 37 / 49 | 18.68source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_medcode.DehpEDgG.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 38.621 \| percent \| reported \| Higher is better \| \| `latency` \| 18.675 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 30000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2 pp sourcePopulation 107 stderr 2.001 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_medcode.DehpEDgG.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:14.899Z Evidence SHA-256 `988a6ae1e07e33ce031572ed4a804be11c36ee97a1dd75cbe4740fff3b534f8c` Source locator `/tasks/overall/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [CorpFin v2 / max fitting context](https://benchize.com/benchmarks/vals-ai%3Acorp_fin_v2%3Amax_fitting_context.md) | 67.13% | SE ±1.6 pp | 10 / 39 | 22.61source duration | [Vals AI · 2](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 67.133 \| percent \| reported \| Higher is better \| \| `latency` \| 22.607 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature Not reported topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.6 pp sourcePopulation 134 stderr 1.604 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_corp_fin_v2.CectKwl1.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 2 Observed 2026-10-22T02:10:13.429Z Evidence SHA-256 `7cf2217ab3728d5a84bffce82aab9fc52d7f810b0cbe8c65b387df47f3d31c9b` Source locator `/tasks/max_fitting_context/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate structured finance securitization](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__structured_finance_securitization.md) | 55.56% | SE ±2.8 pp | 43 / 49 | 2,100.3source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 55.556 \| percent \| reported \| Higher is better \| \| `latency` \| 2100.302 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.8 pp sourcePopulation 81 stderr 2.804 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__structured_finance_securitization/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / corporate governance](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acorporate_governance.md) | 0% | SE ±0 pp | 11 / 49 | 1,552.81source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 1552.808 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/corporate_governance/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / international trade sanctions](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Ainternational_trade_sanctions.md) | 0% | SE ±0 pp | 4 / 49 | 1,963.4source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 1963.404 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/international_trade_sanctions/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / emerging companies venture capital](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Aemerging_companies_venture_capital.md) | 0% | SE ±0 pp | 18 / 49 | 2,558.91source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 0 \| percent \| reported \| Higher is better \| \| `latency` \| 2558.912 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0 pp sourcePopulation 81 stderr 0 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/emerging_companies_venture_capital/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate capital markets](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__capital_markets.md) | 54.9% | SE ±3.12 pp | 45 / 49 | 975.16source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 54.902 \| percent \| reported \| Higher is better \| \| `latency` \| 975.162 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.12 pp sourcePopulation 81 stderr 3.122 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__capital_markets/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate antitrust competition](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__antitrust_competition.md) | 70.29% | SE ±2.76 pp | 40 / 49 | 2,430.91source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 70.29 \| percent \| reported \| Higher is better \| \| `latency` \| 2430.912 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.76 pp sourcePopulation 81 stderr 2.756 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__antitrust_competition/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Legal Research Bench / weighted pass administrative regulatory](https://benchize.com/benchmarks/vals-ai%3Alegal_research%3Aweighted_pass__administrative_regulatory.md) | 72.4% | SE ±2.88 pp | 42 / 50 | 476.59source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.398 \| percent \| reported \| Higher is better \| \| `latency` \| 476.589 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.88 pp sourcePopulation 82 stderr 2.881 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_research.DiuL0R8P.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:07.934Z Evidence SHA-256 `26c58dfd47d0d2dc86f33c6d056f30309e0ee63570e7cd752c593b385f9d862e` Source locator `/tasks/weighted_pass__administrative_regulatory/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | | [Harvey's Legal Agent Benchmark v1 / criteria pass rate banking finance](https://benchize.com/benchmarks/vals-ai%3Ahlab-v1%3Acriteria_pass_rate__banking_finance.md) | 84.02% | SE ±2.24 pp | 36 / 49 | 2,380.22source duration | [Vals AI · 1](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 84.015 \| percent \| reported \| Higher is better \| \| `latency` \| 2380.224 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens 128000 provider Nvidia reasoning Not reported Reasoning effort unknown Temperature 1 topP 0.95 verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.24 pp sourcePopulation 81 stderr 2.239 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_legal_agent_benchmark_v1.ZWRzZKcv.json) Source Vals AI Original model label nvidia/nemotron-3-ultra-550b-a55b Source version 1 Observed 2026-10-22T02:10:09.422Z Evidence SHA-256 `65e7cd51e83066a42c9d782635dba5f2e9e1b189bd8118d3510cd17d2ddcba3f` Source locator `/tasks/criteria_pass_rate__banking_finance/nvidia~1nemotron-3-ultra-550b-a55b` | | | | | | | [Next page](https://benchize.com/models/nemotron-3-ultra-550b-a55b.md?cursor=add36d9d6fc112234c6af515fcd160b282e083688d2149808ec8cf3a420e76ec) --- Cite as: Nemotron 3 Ultra — Benchize, https://benchize.com/models/nemotron-3-ultra-550b-a55b (retrieved 2026-10-22)