Multimodal benchmarks

Human pairwise evaluation of svg outputs, measuring preference. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with 95%…

Human pairwise evaluation of svg outputs, measuring alignment. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with 95%…

Human pairwise evaluation of svg outputs, measuring coherence. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with 95%…

Human pairwise evaluation of svg outputs, measuring overall preference. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with…

Human pairwise preferences on the complete prompt population in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference…

Human pairwise preferences on the complete prompt population in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting…

Humans perceive far more in a scene than what is explicitly depicted: a single glance captures past causes and future trajectories; a quick peek determines if a vehicle can fit…

Detecting assembly mistakes from photographs of three IKEA furniture builds and their official assembly manuals. Models judge whether the build is correct and identify and explain…

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Resolving GitHub issues in JavaScript front-end repositories where the report is a screenshot as much as a description

Multimodal understanding on MMMU-Pro without Python augmentation.

Human pairwise evaluation of llm-to-3d outputs, measuring preference. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with…

Human pairwise evaluation of llm-to-3d outputs, measuring alignment. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with 95%…

Human pairwise evaluation of llm-to-3d outputs, measuring coherence. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with 95%…

Human pairwise evaluation of llm-to-3d outputs, measuring creativity. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with…

Human pairwise evaluation of llm-to-3d outputs, measuring overall preference. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale…

Human pairwise preference evaluation of image generation. Voters compare model outputs for the same request; published Elo ratings and separate model configurations are retained…

Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy.

Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy.

Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy.

Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy.

Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy.

Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy.

Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy.

Benchmark consisting of a weighted performance across finance, coding, and education tasks. Showing the potential impact that LLM's can have on the economy.

Human pairwise evaluation of text-to-image outputs, measuring text coherence. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale…

Human pairwise evaluation of text-to-image outputs, measuring coherence. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with…

Human pairwise evaluation of text-to-image outputs, measuring preference. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale…

Human pairwise evaluation of text-to-image outputs, measuring alignment. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with…

Human pairwise evaluation of text-to-image outputs, measuring overall preference. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo…

Atomic visual perception across the ten capabilities defined by PerceptionBench.

Human pairwise preference evaluation of audio realism. Voters compare model outputs for the same request; published Elo ratings and separate model configurations are retained for…

Human pairwise preference evaluation of image editing. Voters compare model outputs for the same request; published Elo ratings and separate model configurations are retained for…

Audio MultiChallenge is a multi-turn benchmark that evaluates conversational intelligence in spoken dialogue systems and audio-language models, including both speech-to-speech and…

Audio MultiChallenge is a multi-turn benchmark that evaluates conversational intelligence in spoken dialogue systems and audio-language models, including both speech-to-speech and…

Fine-grained visual perception and reasoning with Python assistance.

Reasoning questions about scientific charts in the CharXiv RQ scope without Python assistance.

CharXiv RQ scientific-chart questions with Python assistance.

Solving visual mathematics problems without Python assistance.

Solving visual mathematics problems with Python assistance.

MMMU-Pro with Python assistance, following the source’s disclosed image-order protocol.

Understanding complex document layouts in the OmniDocBench evaluation.

Visual world-knowledge questions evaluated under the source’s forced-answer protocol.

ZeroBench main-set pass@5 over five runs without Python assistance.

ZeroBench main-set pass@5 over five runs with Python assistance.

Human pairwise preference evaluation of text to speech. Voters compare model outputs for the same request; published Elo ratings and separate model configurations are retained for…

VisualToolBench (VTB)is the first benchmark designed to evaluate how well Multimodal Large Language Models (MLLMs) can dynamically interact with and reason about visual…

Human pairwise evaluation of physics outputs, measuring with-reference. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with…

Human pairwise evaluation of physics outputs, measuring looks-like-reference-pinned. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s…

Human pairwise evaluation of physics outputs, measuring with-reference-pinned. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo…

Human pairwise evaluation of physics outputs, measuring blind. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale with 95%…

Human pairwise evaluation of physics outputs, measuring overall preference. Scores are fitted using the Bradley–Terry preference model and expressed on the publisher’s Elo scale…

The growing capabilities of multimodal language models have created two critical challenges in their evaluation. First, as these models become more prevalent, their potential…

Human pairwise preference evaluation of video generation. Voters compare model outputs for the same request; published Elo ratings and separate model configurations are retained…

Human pairwise preference evaluation of image to video. Voters compare model outputs for the same request; published Elo ratings and separate model configurations are retained for…

Human pairwise preference evaluation of multiple images to video. Voters compare model outputs for the same request; published Elo ratings and separate model configurations are…

Human pairwise preference evaluation of multimodal video generation. Voters compare model outputs for the same request; published Elo ratings and separate model configurations are…

Human pairwise preferences on english prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference ratings rather…

Human pairwise preferences on english prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting for response…

Human pairwise preferences on diagram prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference ratings rather…

Human pairwise preferences on ocr prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference ratings rather than…

Human pairwise preferences on diagram prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting for response…

Human pairwise preferences on ocr prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting for response style;…

Human pairwise preferences on homework prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference ratings rather…

Human pairwise preferences on homework prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting for response…

Human pairwise preferences on creative writing vision prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference…

Human pairwise preferences on creative writing vision prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting…

Human pairwise preferences on chinese prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference ratings rather…

Human pairwise preferences on chinese prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting for response…

Human pairwise preferences on humor prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference ratings rather than…

Human pairwise preferences on humor prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting for response…

Human pairwise preferences on entity recognition prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference ratings…

Human pairwise preferences on entity recognition prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting for…

Human pairwise preferences on captioning prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference ratings rather…

Human pairwise preferences on captioning prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting for response…

Human pairwise preferences on creative writing prompts in Arena’s vision evaluation. The original organizer publishes model ratings and uncertainty; these are preference ratings…

Human pairwise preferences on creative writing prompts in Arena’s vision style control evaluation. The original organizer publishes model ratings and uncertainty, adjusting for…

Can speech recognition preserve exact structured values in workplace audio?

Can speech recognition preserve exact structured values in workplace audio?

Can speech recognition preserve exact structured values in workplace audio?

Human pairwise preference evaluation of video editing. Voters compare model outputs for the same request; published Elo ratings and separate model configurations are retained for…