# Claude 3.5 Sonnet > Source: Benchize (https://benchize.com) — https://benchize.com/models/claude-3-5-sonnet. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/models/claude-3-5-sonnet. Anthropic Canonical URL: https://benchize.com/models/claude-3-5-sonnet Last updated: Thu, 22 Oct 2026 07:32:33 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Anthropic ## Benchmarks | Benchmark | Score | ± | Rank | Cost | Source | | | --- | --- | --- | --- | --- | --- | --- | | [SWE-bench Multimodal](https://benchize.com/benchmarks/swe-bench%3Amultimodal.md) | 25.34% | — | 8 / 14 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 25.34 \| percent \| multimodal \| Higher is better \| ### Configuration agent Agentless Lite agentOrganization Agentless models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250226_agentless_lite_claude3.5 subset Multimodal system ["System: Attempts - 1"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/4/results/11` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 23% | — | 19 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 23 \| percent \| lite \| Higher is better \| ### Configuration agent SWE-agent agentOrganization SWE-agent models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20240620_sweagent_claude3.5sonnet subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/69` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 47.33% | — | 4 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 47.33 \| percent \| lite \| Higher is better \| ### Configuration agent devlo agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241122_devlo subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/14` | | | | | | | | [SWE-bench Verified](https://benchize.com/benchmarks/swe-bench%3Averified.md) | 39.6% | — | 51 / 63 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 39.6 \| percent \| verified \| Higher is better \| ### Configuration agent EPAM AI/Run Developer Agent agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241029_epam-ai-run-claude-3-5-sonnet subset Verified system ["System: Attempts - 2+"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/2/results/138` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 51.33% | — | 1 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 51.33 \| percent \| lite \| Higher is better \| ### Configuration agent Isea agentOrganization Not reported models ["claude-3.5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250911_isea_claude-3.5-sonnet-20241022 subset Lite system ["System: Attempts - 2+"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/6` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 38.33% | — | 14 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 38.33 \| percent \| lite \| Higher is better \| ### Configuration agent Moatless Tools agentOrganization Moatless Tools models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241117_moatless_claude-3.5-sonnet-20241022 subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/32` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 42.67% | — | 8 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 42.67 \| percent \| lite \| Higher is better \| ### Configuration agent Lingxi agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250509_Lingxi_claude-3-5-sonnet-20241022 subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/21` | | | | | | | | [BuzzBench](https://benchize.com/benchmarks/eq-bench%3Abuzzbench.md) | 61.94% | CI 61.07–62.8% | 4 / 15 | — | [eq bench · buzzbench](https://eqbench.com/buzzbench.js) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 61.94 \| percent \| overall \| Higher is better \| ### Configuration interactiveAdjustments none protocol buzzbench sourceModelVersion claude-3.5-sonnet ### Reported statistics Uncertainty Confidence interval: 61.07–62.8% confidenceIntervalHigh 62.8 confidenceIntervalLow 61.07 ### Source [Original source](https://eqbench.com/buzzbench.js) Source eq bench Original model label claude-3.5-sonnet Source version buzzbench Observed 2026-10-22T05:10:06.154Z Evidence SHA-256 `4d0f19f1f87e08772b741a28b1ac39111587b832035dd1a783af3de67d51be1c` Source locator `/leaderboardDataBuzzbench/0` | | | | | | | | [SWE-bench Verified](https://benchize.com/benchmarks/swe-bench%3Averified.md) | 50.8% | — | 44 / 63 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 50.8 \| percent \| verified \| Higher is better \| ### Configuration agent Agentless-1.5 agentOrganization Agentless models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241202_agentless-1.5_claude-3.5-sonnet-20241022 subset Verified system ["System: Attempts - 2+"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/2/results/107` | | | | | | | | [SWE-bench Verified](https://benchize.com/benchmarks/swe-bench%3Averified.md) | 63.4% | — | 22 / 63 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 63.4 \| percent \| verified \| Higher is better \| ### Configuration agent AgentScope agentOrganization Alibaba models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250206_agentscope subset Verified system ["System: Attempts - 2+"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/2/results/66` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 39% | — | 13 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 39 \| percent \| lite \| Higher is better \| ### Configuration agent Moatless Tools agentOrganization Moatless Tools models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250114_moatless_claude-3.5-sonnet-20241022 subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/30` | | | | | | | | [AIME 2025](https://benchize.com/benchmarks/aime-2025.md) | 3.33% | 95% CI 0.12–6.54% | 36 / 36 | 0.009usd per problem | [matharena · aime--aime_2025](https://matharena.ai/competition_tables/aime--aime_2025) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 3.33 \| percent \| competition \| Higher is better \| \| `cost_per_problem` \| 0.009 \| usd_per_problem \| competition \| Lower is better \| ### Configuration competition aime--aime_2025 predicted false sourceModelName Claude-3.5-Sonnet sourceModelVersion anthropic/claude-35 ### Reported statistics Uncertainty 95% confidence interval: 0.12–6.54% confidenceInterval95 3.21 problems 30 ### Source [Original source](https://matharena.ai/competition_tables/aime--aime_2025) Source matharena Original model label Claude-3.5-Sonnet Source version aime--aime_2025 Observed 2026-10-22T05:30:11.892Z Evidence SHA-256 `fab0abcecc4032852ef2cbd2a84d34ffc42202be358b2994bfaba44ae7a0cd9a` Source locator `/table/rows/60` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 41.33% | — | 9 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 41.33 \| percent \| lite \| Higher is better \| ### Configuration agent PatchKitty-0.9 agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241220_PatchKitty-0.9_claude-3.5-sonnet-20241022 subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/24` | | | | | | | | [Longform Creative Writing](https://benchize.com/benchmarks/eq-bench%3Alongform-writing.md) | 58.4% | — | 57 / 93 | — | [eq bench · longform-writing](https://eqbench.com/creative_writing_longform.js) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `overall_score_100` \| 58.4 \| percent \| overall \| Higher is better \| \| `chapter1_avg` \| 14.85 \| rubric_score \| overall \| Higher is better \| \| `chapter2_avg` \| 14.49 \| rubric_score \| overall \| Higher is better \| \| `chapter3_avg` \| 13.74 \| rubric_score \| overall \| Higher is better \| \| `chapter4_avg` \| 13.25 \| rubric_score \| overall \| Higher is better \| \| `chapter5_avg` \| 12.43 \| rubric_score \| overall \| Higher is better \| \| `chapter6_avg` \| 11.22 \| rubric_score \| overall \| Higher is better \| \| `chapter7_avg` \| 9.74 \| rubric_score \| overall \| Higher is better \| \| `chapter8_avg` \| 9.81 \| rubric_score \| overall \| Higher is better \| \| `final_judgement_avg` \| 10.77 \| rubric_score \| overall \| Higher is better \| ### Configuration interactiveAdjustments none protocol longform-writing sourceModelVersion claude-3.5-sonnet ### Source [Original source](https://eqbench.com/creative_writing_longform.js) Source eq bench Original model label claude-3.5-sonnet Source version longform-writing Observed 2026-10-22T05:10:05.903Z Evidence SHA-256 `3ba283ce357ac7620e16db640a961879c30fe3ddc50a27de9a8ed4290784825c` Source locator `/leaderboardDataLongformV3/32` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 40.67% | — | 11 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 40.67 \| percent \| lite \| Higher is better \| ### Configuration agent Agentless-1.5 agentOrganization Agentless models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241202_agentless-1.5_claude-3.5-sonnet-20241022 subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/27` | | | | | | | | [SWE-bench Test](https://benchize.com/benchmarks/swe-bench%3Atest.md) | 18.13% | — | 2 / 4 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 18.13 \| percent \| test \| Higher is better \| ### Configuration agent SWE-agent agentOrganization SWE-agent models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20240620_sweagent_claude3.5sonnet subset Test system ["System: Attempts - 1"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/1/results/12` | | | | | | | | [SWE-bench Verified](https://benchize.com/benchmarks/swe-bench%3Averified.md) | 51.8% | — | 42 / 63 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 51.8 \| percent \| verified \| Higher is better \| ### Configuration agent Engine Labs agentOrganization Engine Labs models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241125_enginelabs subset Verified system ["System: Attempts - 1"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/2/results/104` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 37% | — | 15 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 37 \| percent \| lite \| Higher is better \| ### Configuration agent Patched.Codes Patchwork agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250104_patched_codes_claude-3.5-sonnet-20241022 subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/34` | | | | | | | | [SWE-bench Verified](https://benchize.com/benchmarks/swe-bench%3Averified.md) | 62.8% | — | 24 / 63 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 62.8 \| percent \| verified \| Higher is better \| ### Configuration agent EPAM AI/Run Developer Agent agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250228_epam-ai-run-claude-3-5-sonnet subset Verified system ["System: Attempts - 2+"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/2/results/70` | | | | | | | | [SWE-bench Verified](https://benchize.com/benchmarks/swe-bench%3Averified.md) | 55.4% | — | 33 / 63 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 55.4 \| percent \| verified \| Higher is better \| ### Configuration agent EPAM AI/Run Developer Agent agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241212_epam-ai-run-claude-3-5-sonnet subset Verified system ["System: Attempts - 2+"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/2/results/89` | | | | | | | | [SVGBench Arena](https://benchize.com/benchmarks/svgbench%3Aarena.md) | 1,295 | — | 118 / 191 | — | [svgbench](https://svgbench.ai/api/leaderboard?shape=2) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `rating` \| 1295 \| rating \| reported \| Higher is better \| ### Configuration Reasoning effort unknown sourceModelId model_anthropic_claude_3_5_sonnet ### Reported statistics coverageCount 27 draws 0 judged 22 losses 19 rank 181 ratingDeviation 46 wins 3 ### Source [Original source](https://svgbench.ai/api/leaderboard?shape=2) Source svgbench Original model label Claude 3.5 Sonnet Observed 2026-10-22T04:30:04.538Z Evidence SHA-256 `5c0a9c869593820d57d2c05f6d416e1b44a740b4835e66c66e9ad8b8edc59e2c` Source locator `/rows/172` | | | | | | | | [SWE-bench Multimodal](https://benchize.com/benchmarks/swe-bench%3Amultimodal.md) | 6.19% | — | 13 / 14 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 6.19 \| percent \| multimodal \| Higher is better \| ### Configuration agent Agentless agentOrganization Agentless models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241006_Agentless_claude3.5 subset Multimodal system ["System: Attempts - 1"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/4/results/18` | | | | | | | | [SWE-bench Verified](https://benchize.com/benchmarks/swe-bench%3Averified.md) | 33.6% | — | 55 / 63 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 33.6 \| percent \| verified \| Higher is better \| ### Configuration agent SWE-agent agentOrganization SWE-agent models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20240620_sweagent_claude3.5sonnet subset Verified system ["System: Attempts - 1"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/2/results/147` | | | | | | | | [Spiral-Bench v1.0](https://benchize.com/benchmarks/eq-bench%3Aspiral-bench-v1.0.md) | 47.7% | CI 47.7–47.7% | 8 / 11 | — | [eq bench · spiral-bench-v1.0](https://eqbench.com/spiral-bench_v1.0.js) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score_norm` \| 47.7 \| percent \| overall \| Higher is better \| ### Configuration interactiveAdjustments none protocol spiral-bench-v1.0 sourceModelVersion claude-3.5-sonnet ### Reported statistics Uncertainty Confidence interval: 47.7–47.7% confidenceIntervalHigh 47.7 confidenceIntervalLow 47.7 ### Source [Original source](https://eqbench.com/spiral-bench_v1.0.js) Source eq bench Original model label claude-3.5-sonnet Source version spiral-bench-v1.0 Observed 2026-10-22T05:10:07.395Z Evidence SHA-256 `d939ed9ba6d42226765fb4acf3e440127472566e1b5c2894d4ca25baa5ae75d2` Source locator `/leaderboardDataDelusion/1` | | | | | | | | [HMMT Feb 2025](https://benchize.com/benchmarks/matharena%3Ahmmt--hmmt_feb_2025.md) | 1.67% | 95% CI -0.62–3.96% | 35 / 35 | 0.008usd per problem | [matharena · hmmt--hmmt_feb_2025](https://matharena.ai/competition_tables/hmmt--hmmt_feb_2025) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 1.67 \| percent \| competition \| Higher is better \| \| `cost_per_problem` \| 0.008 \| usd_per_problem \| competition \| Lower is better \| ### Configuration competition hmmt--hmmt_feb_2025 predicted false sourceModelName Claude-3.5-Sonnet sourceModelVersion anthropic/claude-35 ### Reported statistics Uncertainty 95% confidence interval: -0.62–3.96% confidenceInterval95 2.29 problems 30 ### Source [Original source](https://matharena.ai/competition_tables/hmmt--hmmt_feb_2025) Source matharena Original model label Claude-3.5-Sonnet Source version hmmt--hmmt_feb_2025 Observed 2026-10-22T05:30:12.126Z Evidence SHA-256 `8554225c26eb7a3c2649943ee7cf6b404ec1f7441b20fbdcfccbf35b9a864864` Source locator `/table/rows/59` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 44.67% | — | 7 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 44.67 \| percent \| lite \| Higher is better \| ### Configuration agent Kodu-v1 agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241207_kodu_sonnet_v1 subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/18` | | | | | | | | [SWE-bench Verified](https://benchize.com/benchmarks/swe-bench%3Averified.md) | 60.2% | — | 27 / 63 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 60.2 \| percent \| verified \| Higher is better \| ### Configuration agent Learn-by-interact agentOrganization Google models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250110_learn_by_interact_claude3.5 subset Verified system ["System: Attempts - 2+"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/2/results/76` | | | | | | | | [Fortress](https://benchize.com/benchmarks/scale%3Afortress.md) | 12.96source score | CI 10.62–15.3 | 27 / 31 | — | [scale](https://labs.scale.com/leaderboard/fortress) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score` \| 12.96 \| source_score \| reported \| Higher is better \| ### Configuration Reasoning effort unknown Model configuration Claude 3.5 Sonnet sourceModelVersion Not reported thinkingEnabled Not reported ### Reported statistics Uncertainty Confidence interval: 10.62–15.3 contaminationWarning ORS score: 21.07 rank 8 reportedConfidenceHalfWidth 2.34 ### Source [Original source](https://labs.scale.com/leaderboard/fortress) Source scale Original model label Claude 3.5 Sonnet Observed 2026-10-22T03:30:12.378Z Evidence SHA-256 `2c005102a7346206e5d31d9249d525620bef39f6d514668ed48398e710616b0f` Source locator `/rows/7` | | | | | | | | [SWE-bench Multimodal](https://benchize.com/benchmarks/swe-bench%3Amultimodal.md) | 5.03% | — | 14 / 14 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 5.03 \| percent \| multimodal \| Higher is better \| ### Configuration agent RAG agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241006_RAG_claude3.5 subset Multimodal system ["System: Attempts - 1"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/4/results/20` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 41% | — | 10 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 41 \| percent \| lite \| Higher is better \| ### Configuration agent OrcaLoca + Agentless-1.5 agentOrganization Agentless models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250113_OrcaLoca subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/26` | | | | | | | | [Spiral-Bench v1.2](https://benchize.com/benchmarks/eq-bench%3Aspiral-bench-v1.2.md) | 42.6% | CI 42.6–42.6% | 7 / 16 | — | [eq bench · spiral-bench-v1.2](https://eqbench.com/spiral-bench.js) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `score_norm` \| 42.6 \| percent \| overall \| Higher is better \| ### Configuration interactiveAdjustments none protocol spiral-bench-v1.2 sourceModelVersion claude-3.5-sonnet ### Reported statistics Uncertainty Confidence interval: 42.6–42.6% confidenceIntervalHigh 42.6 confidenceIntervalLow 42.6 ### Source [Original source](https://eqbench.com/spiral-bench.js) Source eq bench Original model label claude-3.5-sonnet Source version spiral-bench-v1.2 Observed 2026-10-22T05:10:07.179Z Evidence SHA-256 `2f0402654b01e7363c0d02bbc708afd7849e669dc5da63d9f85c995ed8f0a97f` Source locator `/leaderboardDataDelusion/1` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 26.67% | — | 18 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 26.67 \| percent \| lite \| Higher is better \| ### Configuration agent Moatless Tools agentOrganization Moatless Tools models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20240623_moatless_claude35sonnet subset Lite system Not reported ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/60` | | | | | | | | [SWE-bench Verified](https://benchize.com/benchmarks/swe-bench%3Averified.md) | 54.2% | — | 35 / 63 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 54.2 \| percent \| verified \| Higher is better \| ### Configuration agent devlo agentOrganization Not reported models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241108_devlo subset Verified system ["System: Attempts - 2+"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/2/results/93` | | | | | | | | [SWE-bench Lite](https://benchize.com/benchmarks/swe-bench%3Alite.md) | 46% | — | 5 / 22 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 46 \| percent \| lite \| Higher is better \| ### Configuration agent KGCompass agentOrganization GLEAM-Lab models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20250619_KGCompass_claude-3.5-sonnet-20241022 subset Lite system ["System: Attempts - 2+"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/3/results/16` | | | | | | | | [SWE-bench Multimodal](https://benchize.com/benchmarks/swe-bench%3Amultimodal.md) | 11.41% | — | 11 / 14 | — | [SWE-bench](https://www.swebench.com/) | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `resolved` \| 11.41 \| percent \| multimodal \| Higher is better \| ### Configuration agent SWE-agent Multimodal agentOrganization SWE-agent models ["claude-3-5-sonnet-20241022"] Reasoning effort Not reported sourceSubmission 20241006_SWE-agent_M_claude3.5 subset Multimodal system ["System: Attempts - 1"] ### Source [Original source](https://www.swebench.com/) Source SWE-bench Original model label Claude 3.5 Sonnet Observed 2026-10-22T02:30:05.773Z Evidence SHA-256 `c862011a4ee1d1a7199fad6ca0905fa5c64f58216f7217ac1eddf37f28788433` Source locator `/leaderboard-data/4/results/16` | | | | | | | --- Cite as: Claude 3.5 Sonnet — Benchize, https://benchize.com/models/claude-3-5-sonnet (retrieved 2026-10-22)