BEIR Benchmark

BEIR (Benchmarking IR) is a heterogeneous benchmark suite for evaluating the zero-shot generalisation of information retrieval models across eighteen diverse datasets spanning domains including biomedical, legal, financial, and Wikipedia text. It measures how well retrieval systems transfer across topic domains and retrieval task types without domain-specific fine-tuning, providing a standardised framework for comparing dense, sparse, and hybrid retrieval approaches.

Semantic Classification

Content

Compositional Relationships (Components)

SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:hasPart ai:TRECCOVIDDataset))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:hasPart ai:NFCorpusDataset))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:hasPart ai:FiQADataset))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:hasPart ai:ScifactDataset))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:hasPart ai:ArguAnaDataset))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:hasPart ai:HotpotQADataset))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:hasPart ai:RelevanceJudgements))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:hasPart ai:EvaluationQuerySet))

Dependency Relationships

SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:requires ai:EmbeddingModel))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:requires ai:BM25Retriever))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:requires ai:RelevanceJudgement))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:requires ai:NDCG10Metric))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:requires ai:DocumentCorpus))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:requires ai:QuerySet))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:requires ai:FAISSIndex))

Capability Relationships

SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:enables ai:ZeroShotGeneralisation))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:enables ai:SemanticSearch))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:enables ai:DensePasageRetrieval))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:enables ai:RetrievalAugmentedGeneration))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:enables ai:HybridSearchEvaluation))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:enables ai:CrossEncoderReranking))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:enables ai:NeuralIRResearch))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:enables ai:ModelComparisonAtScale))

Implementation Relationships

SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:implements ai:ZeroShotEvaluationProtocol))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:implements ai:NormalisedDiscountedCumulativeGain))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:implements ai:HeterogeneousDomainTesting))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:implements ai:ReproducibleRetrievalEvaluation))

Reduction Relationships

SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:reducesTo ai:InformationRetrieval))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:reducesTo ai:EvaluationBenchmark))
SubClassOf(ai:MTEBBenchmark
  ObjectSomeValuesFrom(ai:hasPart ai:BEIRBenchmark))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:reducesTo ai:ZeroShotBenchmark))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:reducesTo ai:NLPEvaluationFramework))

Support Relationships

SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:supports ai:RAGPipelineDevelopment))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:supports ai:NeuralRetrievalResearch))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:supports ai:ModelComparison))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:supports ai:BenchmarkDrivenDevelopment))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:supports ai:EmbeddingModelSelection))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:supports ai:HybridRetrievalDesign))
SubClassOf(ai:BEIRBenchmark
  ObjectSomeValuesFrom(ai:supports ai:DomainAdaptationResearch))

About

• BEIR (Benchmarking Information Retrieval) arose from a practical crisis in the neural retrieval community. By 2020-2021, the release of Dense Passage Retrieval (DPR; Karpukhin et al. 2020) and related dense neural encoders had demonstrated that transformer-based bi-encoders could substantially outperform classical BM25 on standard benchmarks such as MS MARCO and Natural Questions. However, practitioners and researchers noted that these models, trained exclusively on MS MARCO’s web question-answering data, performed poorly or inconsistently when applied to out-of-domain corpora — medical literature, legal documents, scientific papers, or debate arguments. The generalisation gap was real but lacked a systematic, publicly reproducible measurement framework. Thakur et al. (2021) — Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych, based at the UKP Lab at TU Darmstadt and the sentence-transformers ecosystem — addressed this by assembling eighteen existing retrieval-oriented datasets into a unified evaluation suite with a consistent preprocessing pipeline, the open-source beir Python library, and standardised metrics.
• The name “heterogeneous benchmark” reflects the deliberate diversity of BEIR’s constituent tasks. Unlike MS MARCO, which comprises web search queries answered by short passages from Bing results, BEIR spans nine distinct retrieval task types: fact verification (FEVER, Climate-FEVER, SciFact), question answering (Natural Questions, HotpotQA, FiQA, MSMARCO), duplicate question finding (Quora Question Pairs), biomedical retrieval (NFCorpus, TREC-COVID), argument retrieval (ArguAna, Touche-2020), entity retrieval (DBPedia), citation prediction (SCIDOCS), tweet retrieval (Signal-1M), and news retrieval (TREC-NEWS). Corpus sizes range from around 5,000 documents (SciFact) to 21 million passages (MS MARCO full corpus), and query counts range from 50 to 10,000, creating varying challenges for both corpus indexing and query understanding. The diversity ensures that no single retrieval strategy dominates across all tasks, making BEIR a genuine stress test for generalisation rather than overfitting to one retrieval paradigm.
• A seminal finding from the original BEIR paper was that BM25 — despite being a decades-old lexical matching algorithm with no neural components — remained competitive with or outperformed many dense retrieval models on out-of-domain BEIR datasets, particularly those with specialised vocabularies (NFCorpus for nutrition science, TREC-COVID for biomedical literature). This was a significant challenge to the prevailing narrative that neural dense retrieval was categorically superior, and motivated substantial follow-on research into models that could combine the lexical precision of sparse retrieval with the semantic flexibility of dense retrieval.

Components and Architecture

• Dataset Collection Pipeline — the beir Python library (open-sourced at github.com/beir-cellar/beir) provides consistent preprocessing: each dataset is delivered as a JSON-lines corpus file, a query file, and a query-relevant-passage mapping (qrels) in TREC format. All datasets use a standard split into test queries for evaluation.
• Retrieval Systems Evaluated — the benchmark was originally evaluated on: (1) sparse lexical models: BM25 (Elasticsearch/Pyserini), DocT5Query (expanded BM25); (2) dense bi-encoder models: DPR, ANCE, TAS-B, SBERT variants; (3) late-interaction models: ColBERT; (4) cross-encoder re-rankers; (5) learned sparse models: SPLADE, uniCOIL.
• Evaluation Metrics — nDCG@10 is the primary metric; it rewards models that place the most relevant documents highest, weighted by a log discount factor for rank position. Recall@100 measures how many of all relevant documents appear in the top 100 results, assessing whether the first-stage retrieval provides sufficient coverage for a re-ranking stage. MRR@10 (mean reciprocal rank) is sometimes reported for question-answering subtasks.
• nDCG@10 Computation — for a ranked list of 10 retrieved documents with relevance grades rᵢ: nDCG@10 = DCG@10 / IDCG@10 where DCG@10 = ∑ᵢ₌₁¹⁰ (2^rᵢ - 1) / log₂(i+1) and IDCG@10 is the DCG of the ideal ranking.
• Indexing Infrastructure — FAISS (Facebook AI Similarity Search) is the standard approximate nearest-neighbour index for dense retrieval experiments; BM25 experiments use Elasticsearch or the Pyserini/Lucene-based retrieval toolkit.
• The Eighteen Datasets — grouped by task type:
  • Question Answering: MS MARCO Dev, Natural Questions, HotpotQA, FiQA-2018
  • Fact Verification: FEVER, Climate-FEVER, SciFact
  • Argument Retrieval: ArguAna, Touche-2020
  • Entity Retrieval: DBPedia Entity
  • Duplicate Detection: Quora Question Pairs
  • Biomedical: TREC-COVID, NFCorpus, BioASQ (optional)
  • Citation Prediction: SCIDOCS
  • News: TREC-NEWS (optional), Signal-1M (optional)

Key Models and Results

• The original BEIR paper (2021) revealed a stark performance pattern. On easy in-domain tasks (MS MARCO Dev), dense models like ANCE and TAS-B significantly outperformed BM25 (e.g. nDCG@10 0.646 vs 0.228 for TAS-B vs BM25 on MSMARCO). On out-of-domain datasets, however, BM25 frequently matched or exceeded dense bi-encoders: BM25 achieved nDCG@10 0.328 on SciFact while ANCE achieved 0.507 (dense winning), but BM25 achieved 0.325 on NFCorpus while ANCE achieved only 0.297 (dense losing). This heterogeneous pattern confirmed that dense model superiority was dataset-dependent and that domain transfer was the critical challenge.
• Models designed to improve BEIR generalisation have driven substantial progress. Key developments:
  • SPLADE (Formal et al. 2021-2022) — a learned sparse model using BERT-based neural term expansion; achieves average BEIR nDCG@10 around 0.507-0.540 across public datasets, substantially outperforming BM25 (typically 0.42-0.44 average) while retaining the lexical matching property that aids out-of-domain transfer.
  • ColBERT v2 (Santhanam et al. 2022) — multi-vector late-interaction dense retrieval; each query and document token is represented by its own embedding, with relevance computed via MaxSim across all token pairs; achieves strong zero-shot performance across BEIR by capturing fine-grained lexical-semantic interactions.
  • E5 / E5-Mistral (Wang et al. 2022, 2024) — contrastively trained text embeddings with instruction-tuned generalisation; E5-Mistral (based on the Mistral 7B LLM backbone) achieves MTEB retrieval average around 56-58 nDCG@10.
  • BGE / BGE-M3 (BAAI, 2023-2024) — FlagEmbedding models supporting multi-lingual dense retrieval; BGE-M3 uses a unified architecture for dense, sparse, and multi-vector (ColBERT-style) retrieval, achieving BEIR-competitive performance across 100+ languages.
  • Gemini Embedding 2 (Google DeepMind, 2026) — leads MTEB retrieval (which includes BEIR datasets) with average nDCG@10 of 68.32 as of April 2026, the current state of the art.
  • Voyage 4 Large (Voyage AI, 2025-2026) — second on MTEB retrieval, optimised for long-document embedding and instruction-following retrieval.
  • NV-Embed-v2 (NVIDIA, 2024) — latent embedding model designed for high-recall retrieval; competitive on BEIR and MTEB.

BEIR in the RAG Ecosystem

• BEIR has become the primary evaluation foundation for Retrieval-Augmented Generation (RAG) system benchmarking. Since 2022-2023, the RAG paradigm — pairing a retriever (typically a dense bi-encoder over a vector store) with a generative Large Language Model reader — has become the dominant architecture for knowledge-intensive NLP applications including enterprise search, document QA, and open-domain question answering. The retrieval component in RAG systems is directly assessed using BEIR-derived metrics.
• FeB4RAG (Wang et al. 2024) constructs a federated retrieval benchmark from sixteen BEIR sub-collections with LLM-graded relevance judgements, specifically targeting the multi-domain RAG setting where queries may span heterogeneous corpora. CUBO (2025) benchmarks self-contained RAG systems running on consumer hardware (16 GB RAM, single device) against BEIR-scale corpora, demonstrating that BEIR’s practical scope extends from cloud-scale enterprise deployments to edge inference.
• The connection between BEIR retriever quality and RAG end-task accuracy is strong but non-linear: improving nDCG@10 from 0.40 to 0.60 on a BEIR-like retrieval task typically translates to a substantial improvement in downstream answer accuracy, but at very high retrieval quality (>0.65 nDCG@10), diminishing returns emerge as reader model capability becomes the bottleneck. This motivates joint training of retriever and reader components (Atlas, REALM, RA-DIT) and re-ranking pipelines using Cross-Encoder Reranking models that improve precision from the top-100 recall set.

MTEB Integration and Expansion

• In 2023, Muennighoff et al. introduced MTEB (Massive Text Embedding Benchmark), which incorporated all BEIR datasets as the retrieval task category of a broader text embedding evaluation covering 56+ tasks across 8 task types: retrieval, classification, clustering, pair classification, reranking, semantic textual similarity (STS), summarisation, and bitext mining. The MTEB HuggingFace leaderboard has become the primary community reference for comparing embedding model performance, with BEIR’s 18 datasets constituting its most-watched retrieval subset.
• MTEB v2 (Enevoldsen et al. 2025, arxiv:2506.21182) extends and maintains MTEB, focusing on long-term usability, dataset integrity, and expanding to new task types and languages. As of June 2025, MTEB v2 has been updated with new datasets addressing long-document retrieval, cross-lingual zero-shot retrieval, and instruction-following retrieval (where the model receives a task description alongside the query).
• Community-contributed extensions beyond BEIR include: BEIR-NL (Dutch BEIR, 14 datasets, 2025 ACL-BUCC), BEIR-PL (Polish BEIR), M-BEIR (multilingual multimodal retrieval including image-text pairs, Wei et al. 2024), and LMEB (Long-horizon Memory Embedding Benchmark, 2025) for evaluating embeddings over long document collections.

Academic Context

• BEIR was introduced by Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych (UKP Lab, TU Darmstadt) at NeurIPS 2021 Datasets and Benchmarks Track. Nils Reimers subsequently joined Hugging Face, where sentence-transformers and BEIR became central to the open-source embedding ecosystem. The paper “Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard” (2023) formalised the BEIR leaderboard infrastructure and provided canonical baseline models.
• Key theoretical contributions surrounding BEIR address the generalisation challenge. Formal et al. (2021, 2022) at NAVER Labs Paris introduced SPLADE, showing that learning sparse representations via masked language modelling could yield better out-of-domain transfer than dense retrievers. Santhanam et al. (Stanford) introduced ColBERT v2 with compressed multi-vector representations that achieved strong BEIR performance. Wang et al. (Microsoft Research) introduced E5 (Embeddings from Bidirectional Encoder Representations), demonstrating that fine-tuning with diverse web data and text-matching tasks substantially improves BEIR generalisation. The instruction-tuned embedding paradigm (INSTRUCTOR, Su et al. 2023; E5-Mistral, Wang et al. 2024) further advances zero-shot transfer by conditioning the encoder on a natural-language description of the retrieval task.
• The “generalisation versus specialisation” tension exposed by BEIR has generated rich theoretical debate. Sur et al. (2022) showed that dense models learn query-document matching functions that are highly dataset-specific, explaining poor transfer; Thakur et al. responded with BEIR-AugSPLADE and data augmentation strategies. The discovery that BM25 was competitive on out-of-domain tasks led to renewed interest in hybrid retrieval — combining BM25’s exact-match lexical precision with dense semantic embeddings — a strategy now universally adopted in production RAG systems via Reciprocal Rank Fusion (RRF) or learned fusion weights.
• SIGIR (ACM Special Interest Group on Information Retrieval) and ECIR (European Conference on Information Retrieval) are the primary publication venues for BEIR-related systems research, with substantial overlap with ACL/EMNLP/NAACL for neural retrieval papers. The TREC (Text REtrieval Conference) track at NIST has independently developed complementary evaluation frameworks for biomedical and news retrieval that contributed datasets to BEIR.

Current Landscape (2026)

• As of mid-2026, BEIR remains the gold-standard benchmark for retrieval generalisation evaluation, though its role has partially shifted from a research frontier to an industry quality gate. Most commercial embedding model providers — OpenAI (text-embedding-3-large), Cohere (Embed v3), Voyage AI (Voyage 3, 4), Google (Gemini Embedding 2), Amazon (Titan Embed v2), NVIDIA (NV-Embed-v2), and BAAI (BGE-M3) — publicly report MTEB/BEIR scores as primary quality metrics in model cards and API documentation.
• The MTEB retrieval leaderboard as of April 2026 shows Gemini Embedding 2 at 68.32 average nDCG@10 across the MTEB retrieval tasks (which include BEIR datasets), a dramatic improvement from the best 2021 baseline (~0.42 for BM25 and 0.45-0.50 for top dense models on average across 18 BEIR datasets). This improvement reflects advances in model scale (7B+ parameter LLM backbones used as encoders), instruction tuning for task-specific retrieval, contrastive pre-training data scale, and the replacement of single-vector bi-encoders with more expressive architectures (multi-vector ColBERT-style, late-interaction, hybrid dense-sparse).
• The hybrid dense-sparse retrieval paradigm — combining BM25 or SPLADE scores with dense embedding scores via Reciprocal Rank Fusion or learned fusion — has become the de facto production standard for enterprise RAG systems, directly validated by BEIR showing that neither approach dominates across all domains. Elastic (Elasticsearch ELSER), Weaviate, Pinecone, and Milvus all support hybrid retrieval out of the box as of 2025.
• Long-context retrieval is an emerging BEIR-adjacent challenge: standard BEIR datasets use 512-token document chunks, but production corpora increasingly contain multi-page documents (contracts, scientific papers, books) that require embedding models with extended context windows. Benchmarks like LMEB (2025) and Loong (2024) address this gap.
• Instruction-following retrieval — where the query is accompanied by a natural-language description of the retrieval task type — has emerged as a significant performance lever: models like E5-Mistral, Gecko (Google), and INSTRUCTOR achieve substantial BEIR gains by conditioning the encoder on the task description, effectively using chain-of-thought-style prompting within the embedding model.

UK Context

• UK academic contributions to information retrieval evaluation are substantial. The University of Glasgow’s Terrier IR platform and the associated information retrieval research group (Keith van Rijsbergen, Mounia Lalmas, Iadh Ounis) developed foundational probabilistic retrieval models underpinning BM25 and its extensions; Robertson and Zaragoza’s “The Probabilistic Relevance Framework: BM25 and Beyond” (2009) originated from Microsoft Research Cambridge (MSRC). The University of Glasgow continues to contribute to evaluation methodology in the SIGIR community.
• Microsoft Research Cambridge (MSRC) has been particularly relevant to the BEIR ecosystem: the Bing search team’s MS MARCO dataset (which BEIR includes as a sub-task) was created at MSRC, and researchers there have contributed substantially to learned sparse retrieval (SPLADE-like approaches) and to evaluation methodology. The Alan Turing Institute in London has research programmes on fair and robust IR evaluation, including bias in BEIR-style retrieval benchmarks and evaluation of retrieval systems in high-stakes settings such as legal document review.
• University College London’s Information Studies department and the UCL Centre for Digital Humanities have used BEIR-style evaluation for domain-specific retrieval in archival science and cultural heritage contexts. The University of Edinburgh’s Institute for Language, Cognition and Computation has active research in cross-lingual retrieval relevant to BEIR-NL and BEIR-multilingual extensions. Sheffield NLP (University of Sheffield) and the Manchester IR group have contributed to low-resource and domain-adapted retrieval research evaluated against BEIR.
• In the Northern English industrial context, NHS Digital (Leeds/London) and the National Institute for Health and Care Research (NIHR) have adopted BEIR-validated retrieval models for clinical literature search, systematic review assistance, and health technology assessment document retrieval — high-stakes applications where zero-shot generalisation is critical as medical literature continually evolves beyond any fixed fine-tuning dataset.
• UK-based startups in the enterprise search and RAG space — including Jina AI (London office), Cohere (London office), and numerous healthcare AI companies — benchmark retrieval components against BEIR as standard product quality practice.

Future Directions (2026-2030)

• Multilingual and cross-lingual BEIR — BEIR-NL, BEIR-PL, and M-BEIR are early steps; a comprehensive multilingual BEIR covering 50+ languages with native (non-translated) queries and corpora will be essential for evaluating global-scale retrieval systems. Models like BGE-M3 and Gecko-multilingual are already designed for this evaluation regime.
• Long-document and long-context retrieval — standard BEIR uses 512-token document chunks; future benchmarks must assess retrieval over full documents (10,000+ tokens) as LLM context windows expand and applications shift from passage-level to document-level retrieval. LMEB and similar efforts point toward this frontier.
• Instruction-following and task-conditioned retrieval — the success of instruction-tuned retrievers (E5-Mistral, INSTRUCTOR, Gecko) suggests that future BEIR-like benchmarks will explicitly condition evaluation on task descriptions, testing whether models correctly adapt their retrieval strategy based on query type metadata.
• Agentic retrieval evaluation — as LLM agents perform multi-step retrieval (query reformulation, iterative retrieval, structured search across multiple corpora), BEIR-style single-query evaluation is insufficient; benchmarks like FRAMES (2024) and TREC-RAG (2024) address multi-hop, multi-source retrieval in agentic workflows.
• Code and multimodal retrieval — extending BEIR to code search (given the scale of code retrieval in software engineering), image-text retrieval (M-BEIR), and audio document retrieval will be necessary as multimodal Large Language Model systems become the primary consumer of retrieved content.
• Dynamic and temporal evaluation — BEIR datasets are static snapshots; real-world retrieval must handle rapidly changing document corpora (news, social media, scientific preprints). Temporal BEIR variants that test retrieval on newly published documents will assess whether models generalise temporally as well as domain-wise.
• Evaluation of RAG end-to-end systems — moving beyond first-stage retrieval nDCG@10 to joint retriever-reader evaluation, where the final answer accuracy depends on both retrieval and generation quality; TREC-RAG and KILT (Knowledge-Intensive Language Tasks) pioneered this direction.

Research and Literature

• 1. Thakur, N., Reimers, N., Rücklé, A., Srivastava, A. and Gurevych, I. (2021). “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” NeurIPS 2021 Datasets and Benchmarks Track. — The founding paper.
• 2. Karpukhin, V., Oguz, B., Min, S. et al. (2020). “Dense Passage Retrieval for Open-Domain Question Answering.” EMNLP 2020. — DPR, the canonical dense retriever baseline for BEIR.
• 3. Robertson, S. and Zaragoza, H. (2009). “The Probabilistic Relevance Framework: BM25 and Beyond.” Foundations and Trends in Information Retrieval. — BM25, the primary sparse BEIR baseline.
• 4. Formal, T., Piwowarski, B. and Clinchant, S. (2021). “SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.” SIGIR 2021. — Learned sparse retrieval achieving strong BEIR transfer.
• 5. Formal, T., Piwowarski, B., Lassance, C. and Clinchant, S. (2022). “From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective.” SIGIR 2022. — SPLADE v2: improved BEIR performance.
• 6. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C. and Zaharia, M. (2022). “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction.” NAACL 2022. — ColBERT v2 on BEIR.
• 7. Xiong, L., Xiong, C., Li, Y. et al. (2021). “Approximate Nearest Neighbor Negative Contrastive Estimation for Dense Text Retrieval.” ICLR 2021. — ANCE: hard-negative training for dense retrieval.
• 8. Reimers, N. and Gurevych, I. (2019). “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” EMNLP 2019. — SBERT, foundational bi-encoder architecture evaluated on BEIR.
• 9. Muennighoff, N., Tazi, N., Magne, L. and Reimers, N. (2023). “MTEB: Massive Text Embedding Benchmark.” EACL 2023. — MTEB incorporates BEIR as its retrieval track.
• 10. Wang, L., Yang, N., Huang, X. et al. (2022). “Text Embeddings by Weakly-Supervised Contrastive Pre-training.” arXiv:2212.03533. — E5 embeddings; strong BEIR performance.
• 11. Wang, L., Yang, N., Huang, X. et al. (2024). “Improving Text Embeddings with Large Language Models.” arXiv:2401.00368. — E5-Mistral: LLM backbone for instruction-tuned BEIR retrieval.
• 12. Su, H., Kasner, Z., Gatt, A. et al. (2023). “One Embedder, Any Task: Instruction-Finetuned Text Embeddings.” ACL 2023 Findings. — INSTRUCTOR: instruction-conditioned embedding for BEIR.
• 13. Chen, J., Xiao, S., Zhang, P. et al. (2024). “BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.” arXiv:2309.07597. — BGE-M3: multilingual BEIR-competitive model.
• 14. Lin, J., Ma, X., Lin, S. et al. (2021). “Pyserini: A Python Toolkit for Reproducible Information Retrieval Research with Sparse and Dense Representations.” SIGIR 2021. — Pyserini: toolkit underlying many BEIR experiments.
• 15. Lewis, P., Perez, E., Piktus, A. et al. (2020). “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” NeurIPS 2020. — RAG: canonical use case of BEIR-validated retrieval.
• 16. Izacard, G., Lewis, P., Lomeli, M. et al. (2022). “Few-Shot Learning with Retrieval Augmented Language Models.” JMLR 2023. — Atlas: joint retriever-reader training for RAG.
• 17. Thakur, N., Reimers, N., Rücklé, A. and Gurevych, I. (2023). “Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard.” arXiv:2306.07471. — BEIR leaderboard and canonical baselines.
• 18. Khattab, O. and Zaharia, M. (2020). “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.” SIGIR 2020. — Original ColBERT.
• 19. Lassance, C. and Clinchant, S. (2022). “SPLATE: Sparse Late Interaction Retrieval.” SIGIR 2024 (extended work). — Combining late interaction and sparse representations on BEIR.
• 20. Wang, F., Liu, J., Jiao, M. et al. (2024). “M-BEIR: A Large-Scale Multimodal Retrieval Benchmark.” arXiv:2402.12176. — Multimodal extension of BEIR.
• 21. Enevoldsen, K., Bhatt, I., Chalkidis, I. et al. (2025). “Maintaining MTEB: Towards Long Term Usability and Reproducibility of Text Embedding Benchmarks.” arXiv:2506.21182. — MTEB v2 maintenance paper.
• 22. Thakur, N. et al. (2025). “BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language.” ACL BUCC 2025. — Multilingual BEIR extension.
• 23. Wang, R., Ma, H., Liu, P. et al. (2024). “FeB4RAG: Evaluating Federated Search in the Context of RAG.” SIGIR 2024. — BEIR-derived federated retrieval benchmark for RAG.
• 24. Nogueira, R. and Cho, K. (2019). “Passage Re-ranking with BERT.” arXiv:1901.04085. — Cross-encoder reranking, widely used post-first-stage in BEIR pipelines.
• 25. Nguyen, T., Rosenberg, M., Song, X. et al. (2016). “MS MARCO: A Human Generated Machine Reading Comprehension Dataset.” NeurIPS 2016 Workshop. — MS MARCO, the origin domain for most BEIR baselines.
• 26. Voorhees, E. M. and Harman, D. (eds.) (2005). “TREC: Experiment and Evaluation in Information Retrieval.” MIT Press. — TREC methodology underlying BEIR’s evaluation design.
• 27. Ailog RAG (2026). “BEIR Benchmark Leaderboard 2025 & 2026: NDCG@10 Scores & Rankings.” https://app.ailog.fr/en/blog/news/beir-benchmark-update — Current leaderboard tracking including Gemini Embedding 2 results.
• 28. Elastic (2025). “Use the BEIR benchmark to evaluate search relevance.” Elasticsearch Labs. https://www.elastic.co/search-labs/blog/evaluating-search-relevance-part-1 — Production deployment guidance using BEIR.

Overview

• BEIR is a heterogeneous benchmark suite comprising eighteen retrieval datasets from nine task types.
• It evaluates zero-shot generalisation of Information Retrieval models across domains including biomedical (TREC-COVID, NFCorpus), legal-adjacent (Climate-FEVER), financial (FiQA-2018), scientific (SciFact, SCIDOCS), encyclopaedic (DBPedia Entity, Natural Questions), argumentative (ArguAna, Touche-2020), and multi-hop (HotpotQA) retrieval.
• The central evaluation metric is nDCG@10, measuring ranking quality in the top-10 retrieved results.
• Released with an open-source Python toolkit (beir, pip-installable) for consistent preprocessing, retrieval, and scoring across all datasets.
• Now incorporated as the retrieval task subset of the MTEB Benchmark, the primary leaderboard for comparing Embedding Model quality as of 2024-2026.
• As of April 2026, Gemini Embedding 2 leads MTEB retrieval (which uses BEIR datasets) at 68.32 average nDCG@10, more than 25 points above the original 2021 BM25 baseline (~0.43).
• The benchmark drove development of key models: SPLADE (learned sparse retrieval), ColBERT v2 (multi-vector late interaction), E5-Mistral (LLM-backbone dense embeddings), and BGE-M3 (multilingual multi-granularity), each specifically designed to improve BEIR scores.
• BEIR’s core finding — that dense models trained on MS MARCO fail to generalise to out-of-domain corpora, sometimes performing worse than BM25 — permanently changed how IR researchers approach model evaluation and training data diversity.
• The BEIR leaderboard (arxiv:2306.07471) and MTEB HuggingFace leaderboard are the community reference points; Gemini Embedding 2, Voyage 4 Large, NV-Embed-v2, BGE-M3, E5-Mistral, and Cohere Embed v3 are the leading models as of June 2026.
• Key related concepts: Semantic Search, Dense Passage Retrieval, Hybrid Search, Retrieval-Augmented Generation, Cross-Encoder Reranking, MTEB Benchmark, Embedding Model, BM25, SPLADE, ColBERT, Natural Language Processing.
• The benchmark is maintained at https://github.com/beir-cellar/beir and its official leaderboard is hosted on HuggingFace as part of the MTEB retrieval track leaderboard.
• BEIR represents one of the most impactful evaluation framework contributions in the neural IR era, cited over 2,500 times and adopted universally in both academic and industrial retrieval research.

Historical Development and Context

• The BEIR benchmark was published in the context of a rapidly evolving neural IR landscape.
• In 2017-2018, BERT (Devlin et al. 2019) transformed NLP, and by 2019-2020 IR researchers were adapting BERT-based models to document retrieval.
• The MonoBERT and MonoT5 cross-encoders (Nogueira et al. 2019-2020) achieved state-of-the-art on MS MARCO by jointly encoding query and document.
• However, cross-encoders are impractical for first-stage retrieval at corpus scale (millions of docs), so bi-encoder dense retrievers (DPR, ANCE, TAS-B) emerged as the scalable neural alternative.
• These dense retrievers proved very strong on MS MARCO but showed worrying fragility in evaluation across other domains, a gap that motivated the BEIR paper.
• The NeurIPS Datasets and Benchmarks Track (launched 2021) provided the ideal venue — BEIR was accepted as one of the inaugural papers, rapidly becoming one of the most cited NeurIPS 2021 papers.
• Within one year of publication, BEIR had become the standard evaluation suite for new retrieval papers at SIGIR 2022, EMNLP 2022, NAACL 2022, and ACL 2022.
• The beir GitHub repository exceeded 2,000 stars within six months and is now used by both academic researchers and commercial ML teams worldwide.
• The close coupling of BEIR to the sentence-transformers ecosystem (Reimers, co-author of BEIR) accelerated adoption: BEIR evaluation was integrated directly into the sentence-transformers documentation and tutorials, reaching the large practitioner community that uses that library.

Model Architecture Trends on BEIR (2021-2026)

• Understanding how model architectures have evolved on BEIR reveals the empirical story of neural IR progress.
• 2021 (BEIR launch): Top models are BM25 (average ~0.43), SBERT (average ~0.45), and TAS-B (average ~0.50). DPR and ANCE underperform BM25 on most out-of-domain datasets. The finding that BM25 beats dense retrievers on many BEIR datasets shocks the community.
• 2022: SPLADE (NAVER Labs) improves sparse retrieval substantially (average ~0.52). ColBERT v2 achieves strong cross-domain performance (~0.50+ average) via late-interaction. Contriever (Izacard et al. 2022, Meta) uses contrastive pre-training without supervision, achieving ~0.50 average on BEIR — showing that unsupervised training on diverse web text improves cross-domain transfer.
• 2023: Instruction-tuned embeddings emerge as the dominant paradigm. INSTRUCTOR (Su et al.), E5 (Wang et al.), and GTE (GenTex Embeddings) achieve averages of 0.52-0.55 by conditioning the encoder on task-type descriptions. The HuggingFace MTEB leaderboard launches and becomes the authoritative ranking, incorporating BEIR as the retrieval subset.
• 2024: LLM-backbone embedding models begin dominating the leaderboard. E5-Mistral (Wang et al. 2024) uses a 7B Mistral LLM fine-tuned for embedding, achieving MTEB retrieval ~0.56-0.58. NV-Embed-v2 (NVIDIA) and Gecko (Google) enter the top-5. BGE-M3 provides multilingual parity. Voyage AI’s Voyage 3 achieves top-tier performance with minimal public technical disclosure.
• 2025-2026: Gemini Embedding 2 (Google DeepMind) leads MTEB retrieval at 68.32 average nDCG@10 as of April 2026, more than 25 points above the original 2021 BM25 baseline. Voyage 4 Large is the second-best public model. The gap between proprietary (Gemini, Voyage) and open-weight (BGE-M3, E5-Mistral) models has widened to approximately 8-12 points, driven by training data scale and the use of large proprietary model backbones.
• Open vs closed: The open-weight model landscape has BGE-M3 and multilingual E5 as the strongest freely available models (MTEB retrieval ~56-58), competitive with 2023-era top proprietary models but lagging 2026 proprietary leaders. For privacy-sensitive applications where data cannot leave the organisation (NHS, legal firms, financial institutions), open models evaluated via BEIR remain the default choice.

BEIR and the Broader Evaluation Ecosystem

• BEIR occupies a specific and well-defined position in the hierarchy of Information Retrieval and Natural Language Processing evaluation frameworks. Understanding its relationship to adjacent benchmarks clarifies both its strengths and limitations.
• MS MARCO vs BEIR: MS MARCO (Microsoft Machine Reading Comprehension) is the dominant in-domain retrieval training and evaluation dataset, containing 8.8 million web passages and ~1 million training queries derived from Bing search logs. MS MARCO Dev (6,980 queries) is included as one of BEIR’s eighteen datasets. The critical difference is that most BEIR-evaluated models are fine-tuned on MS MARCO’s training split, so their performance on MS MARCO Dev within BEIR measures in-domain generalisation rather than true zero-shot transfer. BEIR’s value lies in the other seventeen datasets, which represent true held-out domains.
• BEIR vs GLUE/SuperGLUE: GLUE (Wang et al. 2018) and SuperGLUE (Wang et al. 2019) benchmark language model understanding via classification and inference tasks over sentence pairs. These benchmarks primarily evaluate Encoder representations rather than retrieval systems; they do not measure document retrieval quality or semantic search capability. BEIR specifically fills the gap between pure NLU benchmarks (GLUE) and full end-task benchmarks (TriviaQA, HotpotQA), focusing on the first-stage retrieval step.
• BEIR vs TREC: The TREC (Text REtrieval Conference) evaluation framework, run annually by NIST since 1992, provides authoritative deep relevance judgements (pooled from multiple systems) with multi-grade relevance labels. TREC datasets are of very high evaluation quality but cover narrow topic domains (news, web, bio-medical) in each year’s track. BEIR aggregates existing TREC datasets (TREC-COVID, TREC-NEWS) alongside non-TREC collections, providing breadth that annual TREC tracks cannot match while accepting the limitation of shallower relevance judgements.
• BEIR vs MTEB: MTEB (Massive Text Embedding Benchmark; Muennighoff et al. 2023) is a superset: it incorporates BEIR’s 18 retrieval datasets as its retrieval category and adds 38+ additional tasks across classification, clustering, semantic textual similarity, reranking, bitext mining, and summarisation. MTEB is the primary leaderboard for Embedding Model comparison in 2024-2026; a model’s MTEB retrieval score is exactly its average performance across the BEIR retrieval datasets, making BEIR and MTEB retrieval interchangeable in current literature.
• BEIR vs KILT: KILT (Knowledge-Intensive Language Tasks; Petroni et al. 2021) evaluates models on knowledge-intensive tasks including fact verification, entity linking, slot filling, and open-domain QA, measuring whether the correct document is retrieved as well as whether the final answer is correct. KILT is more demanding than BEIR because it requires end-to-end success (correct retrieval followed by correct answer extraction), whereas BEIR measures only retrieval quality. KILT tasks (FEVER, HotpotQA) overlap with BEIR but evaluate a joint retrieval-extraction pipeline.
• BEIR vs RAG benchmarks: TREC-RAG (2024), FRAMES (Falsification, Retrieval, And Multi-hop Evaluation Suite, Google, 2024), and similar benchmarks evaluate full RAG pipelines including multi-hop reasoning, citation accuracy, and answer faithfulness — capabilities that go beyond what BEIR’s nDCG@10 metric measures. BEIR remains the retrieval-component benchmark even in RAG-centric evaluation stacks: practitioners first validate retriever quality with BEIR/MTEB, then validate the full pipeline with FRAMES or TREC-RAG.

Detailed Dataset Profiles

• BEIR’s eighteen datasets span fundamentally different retrieval challenges. Understanding each dataset’s properties is essential for interpreting per-dataset nDCG@10 scores, which vary enormously across models and retrieval paradigms.
• MS MARCO Dev — 6,980 queries drawn from Bing web search logs; 8.8 million passages; human-annotated with sparse relevance judgements (typically one relevant passage per query); the training domain for most BEIR-evaluated models, so in-domain performance on MS MARCO is inflated relative to true zero-shot capability. Dense models perform best here (BM25 nDCG@10 ≈ 0.228; top dense models >0.65).
• TREC-COVID (Voorhees et al. 2021) — 50 queries about COVID-19 research; 171,332 biomedical papers from CORD-19; expert-annotated relevance judgements with binary and graded labels. Dense models trained on web data struggle because biomedical entity vocabularies and query types (clinical, mechanistic, epidemiological) differ sharply from MS MARCO. BM25 achieves nDCG@10 ≈ 0.656, competitive with many dense retrievers.
• NFCorpus (Boteva et al. 2016) — 3,237 queries derived from NutritionFacts.org; 3,633 medical documents; multi-label relevance annotations. Queries are very short (1-2 words) compared to documents (full medical abstracts); this vocabulary mismatch challenges dense bi-encoders. BM25 is exceptionally strong here (nDCG@10 ≈ 0.325) compared to many dense models.
• Natural Questions (NQ) (Kwiatkowski et al. 2019) — 3,452 test queries from Google search; 2.68M Wikipedia passages; open-domain question answering. Dense models trained with NQ data have an advantage; the out-of-domain challenge is modest compared to biomedical datasets.
• HotpotQA (Yang et al. 2018) — 7,405 questions requiring multi-hop reasoning across two Wikipedia passages; the multi-document nature of relevance is unusual for single-stage retrieval. Models must retrieve multiple passages; standard single-vector retrieval is a limiting factor, motivating multi-hop and iterative retrieval approaches.
• FiQA-2018 (Maia et al. 2018) — 648 financial opinion questions from StackExchange; 57,638 financial document passages. Domain shift to finance reveals models’ inability to transfer web-trained semantic understanding to financial jargon and investment analysis discourse.
• ArguAna (Wachsmuth et al. 2018) — 1,406 queries where each query is one argument and the relevant passage is a counter-argument; 8,674 passages. The retrieval task is argument matching (find the best counter-argument), not standard fact retrieval; this unusual task type exposes the domain inflexibility of models pre-trained on factoid QA.
• Touche-2020 (Bondarenko et al. 2020) — 49 controversial question queries (e.g. “Is climate change real?”); 382,545 web page passages; relevance judged by argument quality and topical relevance. Models must retrieve high-quality argumentative content rather than factual passages. BM25 is competitive due to keyword overlap in argumentative text.
• DBPedia Entity (Hasibi et al. 2017) — 400 entity-seeking queries; 4.6M DBPedia entity descriptions. Entity retrieval differs from passage retrieval: the relevant document for “Albert Einstein” is a structured entity description, not a factoid passage. Neural models that understand entity names and descriptions outperform BM25 on most queries.
• SCIDOCS (Cohan et al. 2020) — 1,000 paper abstract queries; 25,657 scientific paper abstracts; relevance defined by co-citation and co-author relationships. The task is finding scientifically related papers, not direct question-answering; semantic embedding models that capture research topic similarity perform well.
• FEVER (Thorne et al. 2018) — 6,666 queries that are factual claims to be verified; 5.4M Wikipedia sentences as the corpus. For each claim, the relevant evidence passages either support or refute it; standard retrieval evaluates whether evidence is found, not whether it confirms or contradicts.
• Climate-FEVER (Diggelmann et al. 2020) — 1,535 climate science fact-checking queries against 5.4M Wikipedia sentences. Domain specificity of climate science (glaciology, atmospheric chemistry) challenges web-trained models.
• SciFact (Wadden et al. 2020) — 300 scientific claim queries; 5,183 scientific paper abstracts; binary relevance (supports/refutes). Scientific language model fine-tuning (BioLinkBERT, SciBERT) substantially improves performance on SciFact; standard MS MARCO-tuned models underperform.
• Quora Question Pairs (QQP) — 10,000 duplicate question detection queries; 522,931 questions. Models retrieve questions semantically similar to the query question; measures paraphrase detection in retrieval form. Dense semantic models excel; BM25 is relatively weak due to paraphrase not sharing keywords.
• Signal-1M (Suarez et al. 2018, optional) — 97 Twitter/news queries; 2.8M news article passages; relevance defined by tweet engagement. The small query set limits statistical reliability.
• TREC-NEWS (Soboroff et al. 2019, optional) — 57 news background link queries; 594,977 news passages. The task is finding background reading for a news article, testing encyclopaedic and contextual retrieval.

Relationship to Retrieval-Augmented Generation

• BEIR’s most significant practical impact has been as the evaluation standard for the retrieval component of Retrieval-Augmented Generation (RAG) systems. The RAG paradigm (Lewis et al. 2020) combines a neural retriever with a generative Large Language Model reader: for each user query, a retriever selects relevant passages from a corpus, and the LLM generates an answer conditioned on those passages alongside the original query. The quality of the final RAG system output is bounded by retrieval quality — if the retriever fails to return the most relevant documents, the LLM cannot compensate regardless of its generation quality.
• The explosion of RAG adoption in 2022-2025 created enormous demand for retrieval system evaluation standards, and BEIR fulfilled this demand precisely. Enterprise RAG developers — building customer-facing question-answering systems over proprietary document corpora — use BEIR-validated embedding models as the default starting point for retrieval component selection. The standard practice is: (1) baseline evaluation on BEIR datasets most similar to the target domain (e.g. FiQA-2018 for financial RAG, TREC-COVID for medical RAG, SciFact for scientific RAG); (2) domain-specific fine-tuning if BEIR baseline performance is insufficient; (3) production deployment with monitoring of proxy retrieval metrics aligned with nDCG@10.
• The connection between retrieval and generation quality in RAG is complex. Izacard & Grave (2020) showed in their Fusion-in-Decoder architecture that LLM readers can aggregate information across many retrieved passages and partially compensate for lower-ranked relevant documents — but only up to a point. Shi et al. (2023) demonstrated in “Lost in the Middle” that LLMs disproportionately use information from the beginning and end of the context window, making the ranking quality of retrieved passages (exactly what nDCG@10 measures) directly relevant to RAG answer quality even when retrieval recall is high. This justifies BEIR’s nDCG@10 metric as a proxy for downstream RAG performance.
• Advanced RAG architectures have extended beyond the basic retrieve-then-read pattern: Iterative RAG (multi-hop retrieval) iteratively reformulates queries based on previously retrieved information, requiring multiple rounds of retrieval; Agentic RAG allows the LLM to decide when to retrieve and what to query; Self-RAG (Asai et al. 2023) trains the LLM to generate special retrieval tokens and to reflect on retrieved evidence. Each of these patterns still relies on a BEIR-validated retriever as its underlying information access component. FeB4RAG (Wang et al. 2024) specifically evaluates federated retrieval across multiple BEIR-derived corpora in multi-hop RAG scenarios.
• Long-context LLMs (Gemini 1.5 at 1M tokens, Claude 3 at 200K tokens) have prompted debate about whether RAG remains necessary when the entire document collection can fit within the model’s context window. In practice, most enterprise corpora (millions of documents, terabytes of text) far exceed any current context window, and latency and cost constraints make in-context retrieval impractical even for smaller corpora. BEIR-validated dense retrievers therefore remain relevant and necessary for large-scale retrieval regardless of LLM context window size.

Retrieval Paradigms Evaluated by BEIR

• BEIR’s most significant contribution is providing a common evaluation ground for three fundamentally different retrieval paradigms, enabling direct comparison under identical conditions.
• Sparse Retrieval uses term-based representations where both queries and documents are represented as high-dimensional sparse vectors over a term vocabulary. Classical BM25 assigns each term a weight combining term frequency (TF), inverse document frequency (IDF), and document length normalisation; queries and documents are represented as TF-IDF-weighted term vectors, and similarity is computed by the dot product of overlapping terms. BM25’s primary advantage is lexical precision: if a query contains a rare technical term (e.g. “CRISPR-Cas9 off-target activity”) that appears in the relevant document, BM25 will rank it highly regardless of semantic context, because exact string matching is strong. Its limitation is that it cannot handle synonymy (a document discussing “cardiovascular events” will not match a query for “heart attacks”) or paraphrase (a document using formal language will not match a colloquial query). On BEIR, BM25’s average nDCG@10 across all 18 datasets is approximately 0.42-0.44; it leads dense models on NFCorpus, TREC-COVID, and argument retrieval datasets. SPLADE (Formal et al. 2021, 2022) extends sparse retrieval with neural term expansion: a Transformer encoder computes term-weight vectors using BERT-style masked language modelling, expanding queries and documents with related vocabulary. SPLADE achieves approximately 0.50-0.54 average nDCG@10 on BEIR, consistently outperforming BM25 while retaining the inverted-index efficiency of sparse retrieval.
• Dense Retrieval uses bi-encoder architectures where a Transformer encoder maps queries and documents to fixed-dimensional dense vectors (typically 768-1024 dimensions), and similarity is measured by cosine distance or dot product. The key advantage is semantic matching: similar meanings map to nearby vectors regardless of surface form differences. Dense models trained on MS MARCO with contrastive learning (TAS-B, ANCE, DPR) achieve strong performance on question-answering BEIR datasets but often underperform BM25 on out-of-domain corpora with specialised vocabulary, as the Embedding Model has not seen the relevant terminology in training. State-of-the-art dense models in 2024-2026 (E5-Mistral, BGE-M3, Gemini Embedding 2, Voyage 4) use LLM backbones with instruction-tuned retrieval, achieving BEIR averages of 60-68 nDCG@10 — a dramatic improvement over the 0.45-0.50 achieved by the best 2021 dense models. The improvement comes from three sources: (1) much larger encoder models (7B parameter LLMs vs. 110M BERT); (2) instruction tuning that tells the encoder what type of retrieval task it is performing; (3) contrastive training on vastly larger and more diverse data collections.
• Multi-Vector (Late Interaction) Retrieval — ColBERT (Khattab & Zaharia 2020) and ColBERT v2 (Santhanam et al. 2022) represent each query and document as a matrix of per-token dense vectors (one embedding per token, not one per document). Relevance is computed via MaxSim: for each query token, find the maximum similarity across all document tokens, then sum these per-token maxima. This late interaction approach is more expressive than single-vector bi-encoders (it captures fine-grained lexical-semantic token matching) while still allowing offline pre-computation of document token matrices, unlike cross-encoders which require a full transformer forward pass per query-document pair. ColBERT v2 achieves consistently strong BEIR performance by combining the semantic expressiveness of neural representations with the lexical matching capability of token-level interaction.
• Cross-Encoder Re-ranking — a cross-encoder takes a query-document pair as joint input to a Transformer, producing a single relevance score after attending across both texts simultaneously. This is the most accurate (but most expensive) retrieval paradigm because the model can consider complex query-document interactions. Cross-encoders are never used for first-stage retrieval over million-document corpora (too expensive), but are universally applied as second-stage re-rankers on the top-100 candidates from a bi-encoder or BM25 first stage. On BEIR, adding a cross-encoder re-ranking stage consistently improves nDCG@10 by 3-8 points over first-stage dense retrieval alone.
• Hybrid Retrieval — combining sparse and dense retrieval signals. Reciprocal Rank Fusion (RRF; Cormack et al. 2009) merges the ranked lists from BM25 and a dense retriever by assigning each document a score of ∑ 1/(k + rank_r(doc)) where k=60 and the sum is over retrievers r. RRF is parameter-free, robust, and consistently outperforms either component on BEIR. Learned score fusion uses a linear or learned combination of BM25 and dense scores, requiring a small labelled validation set. Hybrid retrieval has become the universal production recommendation based on BEIR evidence.

Evaluation Methodology and Reproducibility

• BEIR’s methodological contribution extends beyond dataset aggregation to the definition of a reproducible evaluation protocol. The beir Python library provides: (1) a unified GenericDataLoader that loads any BEIR dataset from a consistent directory structure; (2) a EvaluateRetrieval class wrapping FAISS-based dense retrieval or BM25-based sparse retrieval; (3) automatic metric computation (nDCG@k, MAP@k, Recall@k, MRR@k) via the pytrec_eval library binding to the reference TREC evaluation tool; (4) standardised preprocessing (lowercasing, punctuation handling) applied consistently across all datasets and models.
• Reproducibility challenges in BEIR include: (1) dataset-specific corpus sizes that require different computational resources (MS MARCO at 8.8M passages vs. SciFact at 5,183 passages span 3 orders of magnitude); (2) relevance judgement density — most BEIR datasets have sparse (binary, one-relevant-document) judgements whereas the original retrieval literature uses dense (multi-grade) judgements; (3) the official BEIR paper evaluated models with FAISS flat (exact) nearest-neighbour search, while production systems use approximate search (HNSW, IVF), introducing small discrepancies.
• The “Brewing BEIR” paper (Thakur et al. 2023) addressed reproducibility by providing: (1) canonical Docker containers with pinned library versions; (2) pre-built FAISS indices for all 18 datasets; (3) reference model checkpoints for BM25, SBERT, and DPR baselines; (4) an official hosted leaderboard on HuggingFace. This infrastructure reduced the barrier for new model submissions from days of computation to hours.
• A subtle but important methodological note: BEIR’s nDCG@10 scores are computed using the test-split queries and their associated relevance judgements, while most BEIR models are not trained on any BEIR data. However, models trained on Natural Questions or MS MARCO — which appear as BEIR sub-tasks — have an implicit advantage on those specific sub-datasets, partially violating the “zero-shot” claim. Pure zero-shot evaluation excludes MS MARCO and NQ from averaging, yielding a stricter cross-domain transfer score.

Practical Deployment Guidance

• For practitioners building Retrieval-Augmented Generation systems, BEIR benchmarking provides actionable model selection guidance. The dominant insight from five years of BEIR evaluation is that no single retrieval paradigm dominates across all domain types: dense bi-encoders lead on question-answering domains (NQ, MS MARCO, FiQA); BM25 leads or ties on technical vocabulary domains (NFCorpus, TREC-COVID); late-interaction models (ColBERT v2) provide the most consistent performance across both domain types; and hybrid dense-sparse fusion with Reciprocal Rank Fusion (RRF) or learned score fusion consistently outperforms either pure approach.
• For production RAG deployments as of 2025-2026, the recommended stack validated against BEIR benchmarks is: (1) a hybrid first-stage retriever combining BM25/SPLADE with a dense bi-encoder (E5-Mistral, BGE-M3, or Gemini Embedding 2) using RRF for score fusion; (2) a cross-encoder re-ranker (MonoT5, BGE reranker, or Cohere Rerank) applied to the top-100 candidates from the hybrid retriever; (3) a reader Large Language Model conditioned on the top-k re-ranked passages. This three-stage pipeline achieves state-of-the-art BEIR performance on most sub-datasets while maintaining sub-second end-to-end latency through FAISS HNSW approximate search.
• Domain adaptation remains the primary practical challenge exposed by BEIR: a model achieving 0.65 nDCG@10 on MS MARCO Dev may achieve only 0.30 on NFCorpus. Practitioners deploying RAG in specialised domains (medical, legal, financial) should: (a) evaluate first-stage retriever quality on domain-specific held-out queries before deployment; (b) consider fine-tuning or adapting an embedding model on domain-specific data; (c) use cross-encoder reranking aggressively, as rerankers generalise better than bi-encoders to new domains; (d) maintain an ongoing evaluation loop re-running BEIR-style metrics against query logs from production traffic.

Industry Adoption and Open-Source Ecosystem

• The BEIR benchmark has driven the development of a mature open-source ecosystem for evaluating and deploying retrieval systems. The beir Python package (github.com/beir-cellar/beir) provides the canonical implementation, with integrated support for BM25 (Elasticsearch, BM25S), dense retrieval (Sentence Transformers, HuggingFace Transformers), and evaluation metrics via pytrec_eval. The package is pip-installable (pip install beir) and provides automated dataset download from HuggingFace Datasets Hub, making BEIR evaluation accessible to practitioners with moderate ML infrastructure.
• Commercial embedding model providers universally report BEIR/MTEB scores as primary quality metrics in their product documentation and API announcements. OpenAI (text-embedding-3-large), Cohere (Embed v3), Google (text-embedding-004, Gecko, Gemini Embedding 2), Voyage AI (Voyage 3, Voyage 4), Amazon (Titan Embed v2), Jina AI (Jina-Embeddings-v3), BAAI (BGE series), and NVIDIA (NV-Embed-v2) all include BEIR/MTEB scores in model cards, making BEIR the lingua franca of commercial embedding API comparison. As of April 2026, the MTEB Retrieval leaderboard hosted on HuggingFace tracks over 200 models, with Gemini Embedding 2 leading at 68.32 average nDCG@10.
• Vector database providers — Pinecone, Weaviate, Milvus, Qdrant, Chroma, and Elasticsearch — use BEIR benchmarks to validate retrieval quality in their integration guides and product documentation. Elastic’s ELSER (Elastic Learned Sparse EncodeR) model, a SPLADE-like sparse neural retrieval model integrated into Elasticsearch, is evaluated against BEIR and positioned explicitly as a BEIR-validated alternative to dense retrieval for general-purpose enterprise search. Weaviate’s hybrid search documentation presents BEIR evaluations showing that their RRF-based hybrid retrieval outperforms either dense or BM25 alone across BEIR datasets.
• Observability and monitoring for production retrieval systems increasingly use BEIR-derived proxy metrics. Arize AI, WhyLabs, and similar ML observability platforms provide dashboards for tracking embedding similarity distributions and retrieval quality proxies that align with BEIR’s nDCG-based evaluation methodology. The ability to compare production retrieval metrics against BEIR baselines provides a calibration reference for detecting distribution shift in deployed RAG systems.
• The transition from BEIR to domain-specific evaluation is a recognised challenge for practitioners. BEIR’s eighteen datasets are effective for general-purpose model selection but may not represent a specific deployment domain well. Best practice from the ML engineering community is: (1) run BEIR evaluation on the most similar publicly available datasets to understand model ranking; (2) collect 100-500 representative domain queries with relevance labels; (3) run the top-3 BEIR models on this domain-specific set; (4) select the model that optimises domain-specific nDCG@10, which may differ from the BEIR ranking. This two-stage evaluation process ensures that BEIR provides an efficient starting filter while domain-specific data provides the final selection signal.

BEIR in Scientific and Medical Information Retrieval

• Scientific and medical retrieval represent two of the most consequential applications of Information Retrieval systems, and BEIR includes four specialised datasets targeting these domains: TREC-COVID, NFCorpus, SciFact, and SCIDOCS. The consistent finding across these four datasets is that models pre-trained on general web text or MS MARCO question-answering fail to transfer well to specialised scientific vocabulary, making scientific information retrieval a persistent benchmark stress test that motivates domain adaptation research.
• Biomedical retrieval presents unique challenges because: (1) terminology is highly specialised (disease mechanisms, drug names, gene symbols, anatomical terminology); (2) queries from clinicians, researchers, and patients differ significantly in vocabulary and specificity; (3) relevance often requires expert knowledge to judge correctly. TREC-COVID illustrates these challenges: a query for “COVID-19 and cardiovascular complications” requires a retrieval model to understand that “SARS-CoV-2 myocarditis” is relevant even though the query terms do not literally appear. Domain-adapted biomedical models such as BioLinkBERT, PubMedBERT, and SapBERT substantially improve over general-purpose dense retrievers on TREC-COVID and NFCorpus, validating the need for domain-specific embedding models in clinical settings.
• Clinical and NHS applications in the UK context illustrate why BEIR generalisation matters at a societal level. NHS Digital’s evidence synthesis teams use systematic search over MEDLINE, EMBASE, and NICE evidence bases to support clinical guideline development (NICE guidance). BEIR-validated retrieval models are increasingly used to automate parts of this systematic search process, where the cost of false negatives (missing a relevant study) is potentially harm to patients. The heterogeneous nature of BEIR’s biomedical datasets provides a reasonable proxy for evaluating retrieval across different clinical question types (diagnostic, therapeutic, epidemiological).
• Academic literature search — the task of finding prior work relevant to a research question — is directly evaluated by SCIDOCS (citation prediction as retrieval proxy). The CORD-19 dataset underlying TREC-COVID was assembled in 48 hours in March 2020 from multiple publisher APIs as an emergency response to the COVID-19 pandemic; its integration into BEIR enabled timely evaluation of whether IR models could assist researchers navigating a rapidly growing literature. By 2022, models fine-tuned on CORD-19 citation data significantly improved upon general-purpose retrievers on TREC-COVID, demonstrating the effectiveness of in-domain adaptation even with limited supervision.
• Legal document retrieval represents another high-stakes application of zero-shot Information Retrieval, though BEIR’s original eighteen datasets do not include a dedicated legal corpus. BEIRLegal (proposed but not official) and COLIEE (Competition on Legal Information Extraction/Entailment) extend the BEIR evaluation philosophy to case law and statute retrieval. The BEIR methodology — testing on held-out domains — is directly applicable to legal retrieval where the training domain (web text, Wikipedia) differs substantially from the evaluation domain (judicial opinions, legislative texts).

Technical Implementation Details

• Running BEIR evaluations requires attention to several technical details that significantly affect reported scores. Understanding these details is essential for fair comparison between systems.
• Document chunking — BEIR datasets provide corpus documents at varying granularities. MS MARCO provides passages already chunked at ~100 tokens; TREC-COVID provides full paper abstracts (~200 tokens); NFCorpus provides variable-length documents. Dense bi-encoders trained on 512-token sequences need documents pre-truncated at that limit; full documents from longer datasets are typically truncated, potentially losing relevant content in later portions. Long-context embedding models (Jina-Embeddings-v3 at 8192 tokens, SFR-Embedding-Mistral, various Voyage models) improve on this by embedding full documents without truncation.
• Batch inference — encoding millions of documents requires efficient batching. GPU memory limits batch sizes (typically 32-512 passages per batch depending on passage length and model size); larger models require smaller batches or gradient checkpointing during encoding. The beir library handles this automatically but practitioners running large models on multi-GPU setups may need to adjust batch sizes, use data parallelism across GPUs, or shard the corpus across devices.
• Index building — the beir evaluation pipeline builds a FAISS flat index (exact nearest-neighbour search) for reproducibility. FAISS flat search is O(n) per query (brute force over all corpus vectors), which is feasible for BEIR’s largest corpora (~21M passages for MS MARCO Dev) on modern GPUs but may take 30-60 minutes. Production systems use FAISS HNSW or IVF-PQ approximate indices that achieve sub-millisecond query latency at the cost of small recall losses (typically <1% nDCG@10 loss versus exact search).
• BM25 implementation — the beir library originally used Elasticsearch for BM25 baselines; the “Brewing BEIR” paper (2023) standardised on Pyserini/Anserini (Lucene-based) for reproducibility. BM25 parameter choices (k1, b) affect scores: the BEIR defaults (k1=0.9, b=0.4 for SPLADE; k1=1.2, b=0.75 for standard BM25) follow Pyserini defaults optimised over web retrieval. Dataset-specific BM25 parameter tuning can improve individual dataset scores but reduces comparability across datasets.
• Relevance threshold — BEIR datasets use varying relevance threshold conventions. Binary datasets (SciFact, ArguAna) treat all judged-relevant documents as grade 1 and unjudged as grade 0. Multi-grade datasets (TREC-COVID uses 0/1/2 grades; FEVER uses 0/1/2/3 grades) compute nDCG using exponential relevance (2^grade - 1). When comparing systems across BEIR papers, it is essential to verify that the same relevance thresholding is applied.
• Negative document handling — BEIR evaluates only on judged documents (those with explicit relevance labels). Unjudged documents are treated as irrelevant, which can underestimate true model performance for models that retrieve genuinely relevant unjudged documents. This “pooling bias” is a known limitation of BEIR and other TREC-style benchmarks; it is mitigated by deep pooling in the original TREC tracks but is unavoidable in BEIR’s re-use of datasets with limited pools.

Connections to Adjacent Ontology Concepts

• BEIR Benchmark connects to a rich graph of Information Retrieval and Natural Language Processing concepts that together constitute the modern neural IR ecosystem.
• Embedding Model — the primary system type evaluated by BEIR; maps text to dense vectors; trained contrastively on query-document pairs. BEIR is the canonical evaluation benchmark for comparing Embedding Model quality across domains.
• BM25 — the baseline sparse retrieval model against which all BEIR results are compared. BM25 was developed at City University London (Robertson & Jones) and is implemented in Lucene/Elasticsearch. On BEIR, BM25 achieves approximately 0.42-0.44 average nDCG@10, remaining competitive with many neural dense retrievers on out-of-domain datasets.
• Dense Passage Retrieval — the original neural dense retrieval approach (DPR, Karpukhin et al. 2020, Facebook/Meta). DPR was one of the primary models evaluated in the original BEIR paper; its poor out-of-domain performance relative to BM25 on several BEIR datasets provided the empirical motivation for the benchmark’s existence.
• Semantic Search — the application capability that BEIR evaluates; finding documents semantically relevant to a query without requiring exact keyword matches. BEIR demonstrates that semantic search quality varies dramatically across domains, motivating domain-adaptive training and hybrid retrieval strategies.
• Retrieval-Augmented Generation — the primary downstream application of BEIR-validated retrieval models; RAG systems pair a BEIR-evaluated retriever with a generative LLM reader. BEIR quality directly proxies for RAG retrieval component quality.
• Cross-Encoder Reranking — the second-stage re-ranking component in production retrieval pipelines; always applied on top of a BEIR-validated first-stage retriever to improve precision from the top-100 candidates retrieved by the bi-encoder or BM25 stage.
• Hybrid Search — the combination of BM25 and dense retrieval via Reciprocal Rank Fusion; BEIR evidence motivates hybrid search as consistently outperforming either alone, particularly in out-of-domain settings. Now the universal production recommendation.
• MTEB Benchmark — the superset benchmark incorporating all 18 BEIR retrieval datasets alongside classification, clustering, STS, and other tasks. MTEB is the primary leaderboard for comparing Embedding Model performance; its retrieval subset is exactly BEIR.
• ColBERT — a multi-vector late-interaction retrieval model achieving strong BEIR performance by computing per-token similarity between query and document tokens (MaxSim operation); bridges dense and sparse retrieval paradigms by retaining token-level granularity.
• SPLADE — the leading learned sparse retrieval model on BEIR; uses neural term expansion to produce sparse, high-dimensional document and query representations; achieves better out-of-domain transfer than standard dense bi-encoders on many BEIR datasets.
• Natural Language Processing — the parent discipline within which BEIR operates; BEIR is a specialised evaluation framework within the NLP evaluation ecosystem, sitting below GLUE/SuperGLUE (general NLU) and above application-specific benchmarks.
• Information Retrieval — the parent sub-discipline directly addressed by BEIR; BEIR represents the application of the IR evaluation paradigm (TREC methodology) to zero-shot generalisation across heterogeneous domains and retrieval task types.
• Vector Database — the infrastructure technology enabling dense retrieval at corpus scale; BEIR experiments use FAISS (Facebook AI Similarity Search) as the reference vector store; production deployments use Pinecone, Weaviate, Milvus, Chroma, or Qdrant.
• Transformer — the underlying architecture of all state-of-the-art BEIR retrieval models; both dense bi-encoders (BERT, E5, BGE, Gemini Embedding) and cross-encoder rerankers (MonoBERT, MonoT5, BGE Reranker) are Transformer-based.
• Question Answering — one of the primary BEIR task types (Natural Questions, HotpotQA, FiQA, MS MARCO); QA retrieval requires finding passages that directly answer specific questions, the task most aligned with MS MARCO training data and most easily mastered by dense retrievers.
• Large Language Model — LLMs now serve as both the backbone for state-of-the-art embedding models (E5-Mistral uses Mistral 7B; Gecko uses Gecko-base; Gemini Embedding 2 uses Gemini architecture) and as the reader component in RAG systems whose retrieval quality BEIR evaluates.

Key Terminology Glossary

• BEIR — Benchmarking Information Retrieval; a heterogeneous zero-shot evaluation benchmark comprising 18 retrieval datasets across 9 task types, introduced at NeurIPS 2021 by Thakur et al.
• BM25 — Best Match 25; a probabilistic sparse retrieval model that scores documents based on term frequency, inverse document frequency, and document length normalisation; the canonical sparse retrieval baseline in BEIR; originated in the Okapi project at City University London.
• Bi-Encoder — a dual-encoder architecture encoding queries and documents independently into dense vectors for similarity search; the standard architecture for dense first-stage retrieval; also called a bi-encoder or dual-encoder.
• ColBERT — Contextualized Late Interaction over BERT; a multi-vector retrieval model in which each token of the query attends to the token most similar across all document tokens (MaxSim); balances the efficiency of bi-encoders with the expressiveness of cross-encoders.
• Contrastive Learning — the training paradigm used for embedding models: a positive query-document pair is pushed together in embedding space while negative pairs are pushed apart; hard negatives (retrieved by BM25 but not relevant) improve training quality.
• Cross-Encoder Reranking — a re-ranking model that takes a query-document pair as joint input to a Transformer, producing a single relevance score; expensive at scale but produces more accurate relevance scores than bi-encoders; used as a second stage after first-stage retrieval in production pipelines.
• Dense Passage Retrieval (DPR) — the original Facebook/Meta dense bi-encoder retrieval model (Karpukhin et al. 2020); trained on Natural Questions with in-batch negatives; the model that proved dense retrieval could outperform BM25 on in-domain QA, motivating BEIR’s cross-domain evaluation.
• FAISS — Facebook AI Similarity Search; an open-source library for efficient similarity search over dense vector collections; provides flat (exact), IVF (inverted file, approximate), and HNSW (hierarchical navigable small world graph) indices; the standard indexing backend for BEIR dense retrieval experiments.
• Hybrid Search — a retrieval strategy that combines sparse (BM25/SPLADE) and dense (bi-encoder) retrieval scores; typically fused via Reciprocal Rank Fusion (RRF) or learned score weighting; consistently achieves better BEIR scores than either approach alone.
• Information Retrieval — the discipline of representing, storing, and providing access to documents so that user information needs expressed as queries can be satisfied; BEIR is the primary zero-shot evaluation benchmark for modern neural IR.
• MTEB — Massive Text Embedding Benchmark; extends BEIR to 56+ tasks across 8 task types (retrieval, classification, clustering, STS, summarisation, etc.); incorporates all 18 BEIR datasets as its retrieval subset; the primary community leaderboard for comparing Embedding Model quality.
• nDCG@10 — Normalised Discounted Cumulative Gain at rank 10; the primary BEIR evaluation metric; measures ranking quality by weighting relevance of retrieved documents by a log discount of their position; normalised to [0,1] against the ideal ranking.
• Embedding Model — a model that maps text (query or document) to a dense vector representation; used in dense retrieval for semantic similarity search; BEIR evaluates bi-encoder embedding models as the primary dense retrieval component.
• Qrels — query-relevance judgements; a mapping from (query_id, document_id) pairs to integer relevance grades; the ground truth data that BEIR uses to compute nDCG@10 and Recall@100.
• Recall@100 — the fraction of all relevant documents for a query that appear in the top-100 retrieved results; measures the coverage of the first-stage retriever before re-ranking; a high Recall@100 is necessary (though not sufficient) for a high-quality RAG pipeline.
• Semantic Search — search based on the semantic meaning of the query and documents rather than keyword matching; enabled by dense embedding models trained with contrastive objectives; BEIR measures the cross-domain quality of semantic search models.
• SPLADE — Sparse Lexical and Expansion model; a learned sparse retrieval model that uses a Transformer encoder to produce sparse, high-dimensional term-weight vectors; combines the interpretability and lexical precision of sparse retrieval with neural term expansion for improved recall; achieves strong BEIR cross-domain transfer.
• Zero-Shot Generalisation — the ability of a model to perform well on tasks or domains not seen during training; the central property measured by BEIR; a model achieving high average nDCG@10 across all 18 BEIR datasets demonstrates strong zero-shot retrieval generalisation.

Limitations and Criticisms of BEIR

• Despite its dominant position in the retrieval evaluation ecosystem, BEIR has received substantive methodological criticism that practitioners should understand before treating nDCG@10 as a proxy for real-world retrieval quality.
• Shallow relevance judgements — most BEIR datasets have binary relevance labels collected for a small pool of documents, not the dense multi-grade pools of TREC tracks. Unjudged documents are treated as irrelevant, which can unfairly penalise models that retrieve genuinely relevant documents outside the judgement pool.
• MS MARCO contamination — both MS MARCO Dev and Natural Questions are included in BEIR while also being the primary training datasets for most BEIR-evaluated models. Evaluating on training-domain data inflates scores for those sub-datasets, biasing overall averages.
• Static snapshot — BEIR datasets were collected at specific points in time; corpora age. The COVID-19 literature in TREC-COVID is now 5+ years old, and queries about recent developments are absent. Static benchmarks systematically underestimate the challenges of real-time information retrieval over evolving corpora.
• English-only — the original 18 BEIR datasets are English-only, excluding the majority of global information retrieval use cases. Multilingual variants (BEIR-NL, BEIR-PL) exist but are not yet standardised into a single multi-BEIR evaluation.
• Text-only — BEIR evaluates only text retrieval; multimodal retrieval (image+text, audio, video) is not covered. M-BEIR (Wei et al. 2024) extends BEIR to multimodal settings.
• No temporal/conversational queries — BEIR queries are single-turn; real-world retrieval often involves multi-turn conversations where context from previous queries affects retrieval intent.
• Small test sets — several BEIR datasets have fewer than 100 test queries (SciFact: 300, ArguAna: 1406, Signal-1M: 97). Small test sets lead to high variance in nDCG@10 estimates, making statistically meaningful comparisons difficult without significance testing.
• Benchmark saturation — as model performance approaches 0.65-0.70 average nDCG@10, the BEIR datasets approach saturation on some sub-tasks, reducing their discriminative power for comparing frontier embedding models. This motivates harder benchmark extensions (LMEB, long-context benchmarks, instruction-following retrieval).
• Despite these limitations, BEIR remains the best available standard for zero-shot retrieval generalisation evaluation. The research community has responded to each criticism constructively: MTEB expands the task coverage; BEIR-NL and BEIR-PL address monolinguality; FeB4RAG addresses multi-hop and conversational evaluation; TREC-RAG addresses end-to-end quality. BEIR’s foundational contribution — establishing that in-domain performance on MS MARCO does not predict cross-domain generalisation — has permanently altered how the IR community evaluates retrieval systems.

Integration with Vector Databases and Production Search Systems

• The BEIR benchmark has directly influenced the feature roadmap of commercial vector database and search infrastructure providers.
• Elasticsearch / ELSER — Elastic developed ELSER (Elastic Learned Sparse EncodeR), a SPLADE-like neural sparse model optimised for Elasticsearch’s inverted-index architecture. ELSER was benchmarked against BEIR to demonstrate that sparse neural retrieval achieves better zero-shot generalisation than BM25 while being deployable within existing Elasticsearch infra.
• Pinecone — the leading commercial vector database provides native BEIR evaluation scripts and reports MTEB scores for recommended embedding models in its documentation, using BEIR generalisation scores to justify multi-index hybrid search recommendations.
• Weaviate — documents hybrid search (BM25 + dense vectors with RRF fusion) using BEIR as the primary evidence base; specifically cites BEIR findings that BM25 outperforms dense models on several out-of-domain datasets as motivation for hybrid retrieval.
• Milvus / Zilliz — the open-source and commercial Milvus vector database platform publishes BEIR evaluation guides and uses BEIR scores to benchmark its embedded embedding model integrations (bgm-m3, e5-large-v2, etc.).
• Cohere Rerank — Cohere’s cross-encoder reranking API is positioned as a BEIR-compatible second-stage component; Cohere publishes experiments showing that adding Cohere Rerank after any dense first-stage retriever improves nDCG@10 across BEIR datasets by 3-8 points.
• Qdrant — the Rust-based vector database provides native hybrid search and benchmarks embedding models against BEIR via its integration with HuggingFace Sentence Transformers.
• The convergence of BEIR as the shared evaluation language across these competing products represents a rare instance of academic benchmark adoption by industry at scale, validating BEIR’s design choices and the importance of the zero-shot generalisation problem.

Provenance