BEIR and Zero-Shot Retrieval Evaluation

BEIR is a collection of retrieval datasets from deliberately unrelated domains — scientific claims, financial questions, argument retrieval, COVID literature, duplicate forum questions — evaluated without any training on the target domain. Its metric, nDCG@10, is worth being able to compute by hand, because the two things that go wrong with a BEIR number are both visible in the arithmetic.

What BEIR is

BEIR — Benchmarking Information Retrieval — was published by Thakur and colleagues in 2021. It gathers a set of existing retrieval datasets, converts them to a common format (a corpus, a set of queries, and relevance judgements linking them), and defines a single evaluation protocol across all of them.

The datasets are chosen to be heterogeneous. Fact verification against Wikipedia, scientific claim verification, question answering, duplicate detection on Quora, argument retrieval, biomedical retrieval, entity-centric search, news. Document lengths range from a sentence to a full article; query lengths from a keyword to a paragraph; corpus sizes from a few thousand documents to millions.

That heterogeneity is the point. A retriever tuned on web search queries and then dropped onto scientific claim verification is doing something it was not prepared for, which is the situation you are in when you point a general model at your own documents.

What zero-shot means here

BEIR’s protocol is that a model may be trained on MS MARCO — a large web-search relevance dataset that is conventionally treated as the in-domain training source — and is then evaluated on the other datasets without any further training, fine-tuning or adaptation on them.

This convention is what makes BEIR scores mean anything, and it is also the convention most easily broken without anyone noticing. A model trained on a corpus that happens to include the target datasets’ training splits, or on synthetic queries generated from those corpora, is not zero-shot in the sense BEIR intends, even if no one ran a fine-tune. Modern embedding models are trained on very large mixed corpora, and few publish the mixture.

nDCG@10, computed by hand

BEIR’s headline metric is normalised discounted cumulative gain at rank 10. It rewards putting relevant documents high, discounts them logarithmically as they fall down the list, and normalises against the best possible ordering so that scores are comparable across queries with different numbers of relevant documents.

                 10        rel_i
DCG@10  =       SUM   ---------------
                i=1     log2(i + 1)

  rel_i = the relevance grade of the document at rank i
          (0/1 for binary judgements, 0..2 or 0..3 for graded ones)

IDCG@10 = the same sum computed over the best possible ordering,
          i.e. the top 10 judgements sorted by grade descending.

nDCG@10 = DCG@10 / IDCG@10        (always between 0 and 1)

The discount weights are worth internalising, because they explain why reranking pays.

rank i:      1      2      3      4      5      6      7      8      9     10
log2(i+1):  1.000  1.585  2.000  2.322  2.585  2.807  3.000  3.170  3.322  3.459
weight:     1.000  0.631  0.500  0.431  0.387  0.356  0.333  0.315  0.301  0.289

Moving one relevant document from rank 4 to rank 1 adds 1.000 - 0.431 = 0.569.
Moving one from rank 10 to rank 9  adds 0.301 - 0.289 = 0.012.
The top three positions carry most of the score.

A full worked query, with binary relevance:

Query has 3 relevant documents. Your system ranks them at positions 1, 4 and 8.

DCG@10  = 1/log2(2) + 1/log2(5) + 1/log2(9)
        = 1/1.000   + 1/2.322   + 1/3.170
        = 1.0000    + 0.4307    + 0.3155
        = 1.7462

Ideal ordering puts all three at ranks 1, 2, 3:
IDCG@10 = 1/log2(2) + 1/log2(3) + 1/log2(4)
        = 1.0000    + 0.6309    + 0.5000
        = 2.1309

nDCG@10 = 1.7462 / 2.1309 = 0.8195

Now rerank so the three sit at 1, 2 and 5:
DCG@10  = 1.0000 + 0.6309 + 0.3869 = 2.0178
nDCG@10 = 2.0178 / 2.1309 = 0.9469

Same three documents retrieved. Seventeen points of nDCG.

That last observation is the entire commercial case for reranking. The retriever found the same documents in both cases; only the order changed, and the metric moved seventeen points. It is also why nDCG@10 alone can be misleading about a first-stage retriever: what a first stage owes you is recall in the top 100, not order, and a cross-encoder supplies the order afterwards. Report recall@100 alongside nDCG@10 if you are choosing a first stage.

The judgement pool problem

This is the part that makes BEIR numbers systematically pessimistic for modern systems, and almost nobody mentions it.

Relevance judgements are expensive, so no dataset judges every document against every query. Instead, a pool is built: run a set of retrieval systems, take the top results from each, and have humans judge that pool. Everything outside the pool is treated as non-relevant by default. This is standard IR practice and it works well as long as the systems being evaluated are similar to the ones that built the pool.

BEIR’s pools were largely constructed from lexical and early-neural systems. A dense retriever that surfaces a genuinely relevant document those systems never returned gets zero credit for it — the document was never judged, so it counts as irrelevant. The more a retriever differs from the pool contributors, the more of this invisible penalty it absorbs.

• The direction of the bias is knowable. It penalises novelty and favours systems that behave like the pool. It never works the other way.
• It does not affect all datasets equally. Datasets with shallow pools and few judgements per query are worse affected than densely judged ones. Judgement counts per query vary by more than an order of magnitude across BEIR.
• It is not fixable from the outside. Correcting it requires new human judgements on the new system’s output, which is the expense the pool was avoiding.

The honest reading: a BEIR nDCG@10 is a lower bound on a modern retriever’s effectiveness, tighter on some datasets than on others, and small differences between systems that retrieve very different documents are not interpretable.

Nobody runs all of BEIR

Several BEIR datasets require a licence, registration or a data-use agreement, so they cannot be redistributed with the benchmark. The consequence is that the commonly reported “BEIR average” is an average over whichever subset the reporter could obtain — typically the openly downloadable ones, but not always the same set.

Two BEIR averages computed over different subsets are not comparable, and the difference can exceed the differences people are trying to detect. A published BEIR average should list its datasets. When it does not, treat it as a rough indicator and compare per-dataset numbers instead.

Which datasets to actually read

The average is the wrong column. What BEIR is genuinely good for is finding the two or three datasets whose shape matches your retrieval problem, and reading those. Shape here means the relationship between query length, document length and what counts as relevant — not the subject matter.

The last row is the practically important one. BEIR’s own results established that a well-tuned lexical baseline remains competitive on several datasets, and it is strongest exactly where vocabulary is specialised. If your corpus is full of part numbers, gene names or internal acronyms, the BEIR datasets that resemble it are the ones telling you to run both retrievers rather than to pick one.

What it is a bad proxy for

• Retrieval on your corpus. Domain, document length, query style and vocabulary all differ, and BEIR’s own thesis is that out-of-domain transfer is unreliable — which applies to transferring from BEIR to you exactly as much as it applies within BEIR.
• Latency and index cost. nDCG says nothing about vector dimension, index size or query time. Those are on the MTEB leaderboard and in your own infrastructure bill.
• Long-document retrieval. Most BEIR corpora are short passages. Retrieval over long documents raises chunking questions the benchmark never asks — see chunking strategies.
• Hybrid systems. BEIR evaluates a retriever. Most production systems are a retriever plus BM25 plus a reranker plus filters, and the interaction is where the quality is — hybrid search covers the failure cases each half fixes for the other.