# Vals Multimodal Index / swebench > Source: Benchize (https://benchize.com) — https://benchize.com/benchmarks/vals-ai%3Avals_multimodal_index%3Aswebench. When citing or reusing this content, credit "Benchize" and link to https://benchize.com/benchmarks/vals-ai%3Avals_multimodal_index%3Aswebench. Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy. Canonical URL: https://benchize.com/benchmarks/vals-ai%3Avals_multimodal_index%3Aswebench Last updated: Wed, 21 Oct 2026 18:05:10 GMT Reproduction and quotation must name Benchize, link to the canonical HTML URL, and retain individual source citations. Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy. Multimodal · Models tested: 20 · Score publishers: Vals AI ## Model results | # | Model | Score | ± | Cost | Latency | | | --- | --- | --- | --- | --- | --- | --- | | 1 | [GPT-5.6 Sol](https://benchize.com/models/gpt-5-6-sol.md) | 99.02% | SE ±0.98 pp | $1.31 | 237.14source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 99.02 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.311279 \| USD \| per_test \| Lower is better \| \| `latency` \| 237.142 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±0.98 pp sourcePopulation 33 stderr 0.976 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label openai/gpt-5.6-sol Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/openai~1gpt-5.6-sol` | | | | | | | | 2 | [Claude Fable 5](https://benchize.com/models/claude-fable-5.md) | 97.06% | SE ±1.67 pp | $2.05 | 441.2source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 97.059 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.046736 \| USD \| per_test \| Lower is better \| \| `latency` \| 441.198 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.67 pp sourcePopulation 33 stderr 1.673 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label anthropic/claude-fable-5 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/anthropic~1claude-fable-5` | | | | | | | | 3 | [Claude Opus 5](https://benchize.com/models/claude-opus-5.md) | 95.1% | SE ±2.14 pp | $1.55 | 665.89source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 95.098 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.554764 \| USD \| per_test \| Lower is better \| \| `latency` \| 665.89 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.14 pp sourcePopulation 33 stderr 2.138 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label anthropic/claude-opus-5 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/anthropic~1claude-opus-5` | | | | | | | | 4 | [GPT-5.6 Luna](https://benchize.com/models/gpt-5-6-luna.md) | 93.14% | SE ±2.5 pp | $0.05 | 184.77source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 93.137 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.051004 \| USD \| per_test \| Lower is better \| \| `latency` \| 184.771 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±2.5 pp sourcePopulation 33 stderr 2.503 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label openai/gpt-5.6-luna Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/openai~1gpt-5.6-luna` | | | | | | | | 5 | [Claude Opus 4.8](https://benchize.com/models/claude-opus-4-8.md) | 89.22% | SE ±3.07 pp | $1.92 | 630.31source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 89.216 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.923224 \| USD \| per_test \| Lower is better \| \| `latency` \| 630.305 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.07 pp sourcePopulation 33 stderr 3.071 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label anthropic/claude-opus-4-8 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/anthropic~1claude-opus-4-8` | | | | | | | | 6 | [Muse Spark 1.2](https://benchize.com/models/muse-spark-1-2.md) | 85.29% | SE ±3.51 pp | $0.64 | 638.55source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 85.294 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.644011 \| USD \| per_test \| Lower is better \| \| `latency` \| 638.55 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.51 pp sourcePopulation 33 stderr 3.507 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label meta/muse_spark_1_2 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/meta~1muse_spark_1_2` | | | | | | | | 7 | [GPT-5.5](https://benchize.com/models/gpt-5-5.md) | 82.6% | SE ±1.7 pp | $1.36 | 426.43source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.6 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.361765 \| USD \| per_test \| Lower is better \| \| `latency` \| 426.425 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±1.7 pp sourcePopulation 33 stderr 1.695 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label openai/gpt-5.5 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/openai~1gpt-5.5` | | | | | | | | 8 | [Claude Opus 4.7](https://benchize.com/models/claude-opus-4-7.md) | 82.35% | SE ±3.78 pp | $3 | 508.65source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 82.353 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 2.999546 \| USD \| per_test \| Lower is better \| \| `latency` \| 508.652 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±3.78 pp sourcePopulation 33 stderr 3.775 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label anthropic/claude-opus-4-7 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/anthropic~1claude-opus-4-7` | | | | | | | | 9 | [Muse Spark 1.1](https://benchize.com/models/muse-spark-1-1.md) | 78.43% | SE ±4.07 pp | $0.35 | 238.11source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 78.431 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.352022 \| USD \| per_test \| Lower is better \| \| `latency` \| 238.11 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.07 pp sourcePopulation 33 stderr 4.072 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label meta/muse_spark_1_1 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/meta~1muse_spark_1_1` | | | | | | | | 10 | [Claude Sonnet 4.6](https://benchize.com/models/claude-sonnet-4-6.md) | 77.45% | SE ±4.14 pp | $1.71 | 633.68source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.451 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.706873 \| USD \| per_test \| Lower is better \| \| `latency` \| 633.682 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.14 pp sourcePopulation 33 stderr 4.138 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label anthropic/claude-sonnet-4-6 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/anthropic~1claude-sonnet-4-6` | | | | | | | | 10 | [Gemini 3.6 Flash](https://benchize.com/models/gemini-3-6-flash.md) | 77.45% | SE ±4.14 pp | $1.38 | 1,340.98source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 77.451 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.380993 \| USD \| per_test \| Lower is better \| \| `latency` \| 1340.979 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.14 pp sourcePopulation 33 stderr 4.138 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label google/gemini-3.6-flash Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/google~1gemini-3.6-flash` | | | | | | | | 12 | [Claude Sonnet 5](https://benchize.com/models/claude-sonnet-5.md) | 75.49% | SE ±4.26 pp | $1.75 | 1,097.12source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.49 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 1.749654 \| USD \| per_test \| Lower is better \| \| `latency` \| 1097.115 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.26 pp sourcePopulation 33 stderr 4.259 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label anthropic/claude-sonnet-5 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/anthropic~1claude-sonnet-5` | | | | | | | | 12 | [Gemini 3.5 Flash](https://benchize.com/models/gemini-3-5-flash.md) | 75.49% | SE ±4.26 pp | $0.95 | 230.39source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 75.49 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.953882 \| USD \| per_test \| Lower is better \| \| `latency` \| 230.39 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.26 pp sourcePopulation 33 stderr 4.259 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label google/gemini-3.5-flash Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/google~1gemini-3.5-flash` | | | | | | | | 14 | [GPT-5.6 Terra](https://benchize.com/models/gpt-5-6-terra.md) | 73.53% | SE ±4.37 pp | $0.29 | 142.06source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 73.529 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.286403 \| USD \| per_test \| Lower is better \| \| `latency` \| 142.056 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.37 pp sourcePopulation 33 stderr 4.368 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label openai/gpt-5.6-terra Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/openai~1gpt-5.6-terra` | | | | | | | | 15 | [Gemini 3.1 Pro Preview](https://benchize.com/models/gemini-3-1-pro-preview.md) | 72.55% | SE ±4.42 pp | $0.82 | 329.2source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 72.549 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.820116 \| USD \| per_test \| Lower is better \| \| `latency` \| 329.202 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.42 pp sourcePopulation 33 stderr 4.419 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label google/gemini-3.1-pro-preview Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/google~1gemini-3.1-pro-preview` | | | | | | | | 16 | [MiniMax M3](https://benchize.com/models/minimax-m3.md) | 69.61% | SE ±4.55 pp | $0.51 | 765.23source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 69.608 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.508278 \| USD \| per_test \| Lower is better \| \| `latency` \| 765.232 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.55 pp sourcePopulation 33 stderr 4.554 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label minimax/MiniMax-M3 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/minimax~1MiniMax-M3` | | | | | | | | 17 | [Gemini 3 Flash Preview](https://benchize.com/models/gemini-3-flash-preview.md) | 68.63% | SE ±4.59 pp | $0.31 | 327.17source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 68.627 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.306186 \| USD \| per_test \| Lower is better \| \| `latency` \| 327.172 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.59 pp sourcePopulation 33 stderr 4.594 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label google/gemini-3-flash-preview Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/google~1gemini-3-flash-preview` | | | | | | | | 18 | [MiMo-V2.5](https://benchize.com/models/mimo-v2-5.md) | 64.71% | SE ±4.73 pp | $0.01 | 284.37source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.706 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.011672 \| USD \| per_test \| Lower is better \| \| `latency` \| 284.365 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.73 pp sourcePopulation 33 stderr 4.732 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label xiaomi/mimo-v2.5 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/xiaomi~1mimo-v2.5` | | | | | | | | 18 | [Mistral Medium 3.5](https://benchize.com/models/mistral-medium-3-5.md) | 64.71% | SE ±4.73 pp | $4.29 | 854.89source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 64.706 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 4.288697 \| USD \| per_test \| Lower is better \| \| `latency` \| 854.891 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.73 pp sourcePopulation 33 stderr 4.732 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label mistralai/mistral-medium-3.5 Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/mistralai~1mistral-medium-3.5` | | | | | | | | 20 | [Gemini 3.1 Flash Lite Preview](https://benchize.com/models/gemini-3-1-flash-lite-preview.md) | 57.84% | SE ±4.89 pp | $0.11 | 136.08source duration | | | ### Original source values \| Metric \| Value \| Unit \| Scope \| Direction \| \| --- \| --- \| --- \| --- \| --- \| \| `accuracy` \| 57.843 \| percent \| reported \| Higher is better \| \| `cost_per_test` \| 0.106331 \| USD \| per_test \| Lower is better \| \| `latency` \| 136.077 \| source_duration \| reported_duration \| Lower is better \| ### Configuration computeEffort Not reported harness Not reported maxOutputTokens Not reported provider Not reported reasoning Not reported Reasoning effort unknown Temperature Not reported topP Not reported verbosity Not reported ### Reported statistics Uncertainty Standard error: ±4.89 pp sourcePopulation 33 stderr 4.889 tokenTotals Not reported ### Source [Original source](https://www.vals.ai/_astro/benchmark_view_vals_multimodal_index.virVpWMs.json) Source Vals AI Original model label google/gemini-3.1-flash-lite-preview Source version 1.2 Observed 2026-10-21T16:45:51.881Z Evidence SHA-256 `077b22575f6e075b7b78157d000ec8f94d7ead409c7dcbcf04887630ed572a2d` Source locator `/tasks/swebench/google~1gemini-3.1-flash-lite-preview` | | | | | | | --- Cite as: Vals Multimodal Index / swebench — Benchize, https://benchize.com/benchmarks/vals-ai%3Avals_multimodal_index%3Aswebench (retrieved 2026-10-21)