Embeddings benchmarks

Retrieval quality in the legal domain across case documents, statutes, legal summarization, and multilingual legal Q&A, with tasks representative of real-world production…

Retrieval quality in German across legal, healthcare, and business domains, with tasks representative of real-world production retrieval demands. A German-language subset of RTEB.…

Retrieval quality in French across legal and general knowledge domains, with tasks representative of real-world production retrieval demands. A French-language subset of RTEB.…

Retrieval quality in the healthcare and medical domain across medical Q&A, healthcare information retrieval, and multilingual medical consultation, with tasks representative of…

Retrieval quality in the financial domain across finance benchmarks, Q&A, financial document retrieval, and corporate governance, with tasks representative of real-world…

Instruction-following retrieval quality, measuring how well models retrieve relevant documents when given detailed natural language instructions alongside queries.

Zero-shot retrieval quality across a heterogeneous set of IR tasks and domains, providing a common framework for comparing NLP-based retrieval models.

Long-context retrieval quality across synthetic and real-world tasks featuring documents of varying length with dispersed target information.

Chemical domain text embedding quality across bitext mining, classification, clustering, pair classification, and retrieval.

Legal document retrieval across case documents, statutes, legal Q&A, and legal summarization in multiple languages.

German text embedding quality across classification, clustering, pair classification, reranking, retrieval, and semantic similarity.

Dutch text embedding quality across classification, clustering, pair classification, multilabel classification, reranking, retrieval, and semantic similarity.

Medical information retrieval across clinical, biomedical, and consumer health domains, spanning retrieval, reranking, and clustering tasks.

Text embedding quality across Indic languages spanning bitext mining, classification, clustering, pair classification, retrieval, reranking, and semantic similarity.

French text embedding quality across classification, clustering, pair classification, reranking, retrieval, and semantic similarity, using high-quality native French datasets.

Russian text embedding quality across classification, clustering, reranking, pair classification, retrieval, and semantic similarity. In v1.1, MIRACLRetrieval and RiaNewsRetrieval…

Text embedding quality across European languages spanning bitext mining, classification, clustering, pair classification, retrieval, reranking, and semantic similarity.

Spanish text embedding quality across classification, clustering, pair classification, reranking, retrieval, and semantic similarity. For discussion on benchmark construction, see…

Vietnamese text embedding quality across retrieval, classification, pair classification, clustering, reranking, and semantic similarity.

Scandinavian text embedding quality covering Danish, Swedish, Norwegian Bokmål, and Nynorsk and spanning classification, clustering, retrieval as well as bitext tasks across…

Persian text embedding quality across classification, clustering, pair classification, reranking, retrieval, semantic similarity, and summarization retrieval. In v2, large…

Reasoning-intensive retrieval quality across real-world queries spanning diverse domains including economics, psychology, mathematics, and coding, drawn from naturally occurring…

Thai text embedding quality across classification, clustering, pair classification, reranking, and retrieval. Tasks are native Thai or high-quality human translations;…

Chinese text embedding quality across retrieval, reranking, pair classification, clustering, classification, and semantic similarity.

Zero-shot retrieval quality in Dutch across the BEIR task suite, created through automated translation of the original English benchmark.

Code retrieval quality across a wide range of popular programming languages, covering code search, text-to-SQL, and code feedback tasks.

Code information retrieval across diverse programming languages and coding tasks, including code search, question answering, and text-to-SQL retrieval.

Audio-only embedding quality across classification, clustering, pair classification, reranking, and retrieval tasks.

Retrieval quality in English across legal, finance, code, and healthcare domains, with tasks representative of real-world production retrieval demands. An English-only subset of…

Retrieval quality in the code domain across algorithmic problems, data science tasks, code evaluation, SQL retrieval, and multilingual code retrieval, with tasks representative of…

Portuguese text embedding quality benchmark across semantic text similarity, classification, reranking and retrieval.

Code embedding and reranking quality across code-to-text, text-to-code, and code-to-code retrieval tasks, using counterfactually rewritten problems in five programming languages…

Korean text embedding quality across classification, clustering, pair classification (NLI), reranking, retrieval, and semantic similarity. Expanded successor to MTEB(kor, v1):…