Hybrid Search: BM25 + Vectors, and Why You Need Both

Hybrid search is usually justified with “keyword search catches what semantic search misses”, which is true and useless. The argument becomes actionable when you can name the query classes each one drops on the floor.

Two different notions of relevance

BM25 scores a document by how often the query’s terms appear in it, discounted by how common those terms are in the corpus and normalised by document length. It has no idea what any word means. Its strength is that it treats a rare string as extremely informative: a term appearing in one document out of a million gets an enormous inverse-document-frequency weight, and that document is going to rank first.

A dense retriever maps query and document into a shared space where proximity is supposed to mean relatedness, and scores by cosine similarity. It has no idea what any specific string is. Its strength is that “how do I stop the subscription” and “cancellation procedure” land near each other with no shared vocabulary at all.

These are not two implementations of one idea; they are two different relevance functions with disjoint blind spots. The BEIR benchmark (Thakur et al., arXiv:2104.08663) made this concrete at scale: across eighteen heterogeneous retrieval datasets, BM25 remained a strong baseline that several well-regarded dense retrievers failed to beat once taken out of the domain they were trained on. Zero-shot generalisation is where lexical matching keeps earning its place.

What vectors fail on

• Exact identifiers. ERR_TLS_CERT_ALTNAME_INVALID, SKU-44821-B, CVE-2024-3094, getUserByEmail. The tokeniser shreds these into fragments and the embedding averages them into something generic. A query for one error code will happily return a chunk about a different error code.
• Negation and near-antonyms. “plans that do not include support” embeds close to “plans that include support”. The two sentences share almost every token and differ by one, and cosine similarity is not a logic engine.
• Rare proper nouns. A person or product name the embedding model never saw during training has no meaningful position, so it contributes noise rather than signal.
• Numbers and versions. “version 3.11” and “version 3.1” are neighbours in embedding space and different in fact. This one silently produces confident wrong answers, because the retrieved chunk looks entirely on-topic.

What BM25 fails on

• Vocabulary mismatch. The user writes “laid off”; the handbook says “involuntary separation”. Zero term overlap, zero score, no result. This is the classic case dense retrieval was built for.
• Paraphrase and question form. Documents are written as statements and queries arrive as questions, and the function words a question adds are exactly the ones BM25 discounts to nothing.
• Multilingual corpora. A Dutch query against English documentation scores zero on every term. A multilingual embedding model handles it as a matter of course.
• Morphology beyond the stemmer. Stemming catches running to run. It does not catch authentication to sign-in.

Read the two lists together and the pattern is clear: BM25 fails when the words differ and the meaning is the same; dense retrieval fails when the meaning is close and the exact string is what matters. Almost every real corpus contains both kinds of query, which is why the answer is both methods rather than a better one of either.

Fusing two rankings

The obvious approach — normalise both score distributions and take a weighted sum — is fragile. BM25 scores are unbounded and corpus-dependent, cosine similarities are squeezed into a narrow band near the top, and min-max normalising over a top-k window means the same document gets a different normalised score depending on what else was retrieved.

Reciprocal rank fusion sidesteps this by throwing the scores away and using only the ranks. It comes from Cormack, Clarke and Buettcher (SIGIR 2009), and the whole method is one line:

from collections import defaultdict

def rrf(rankings, k=60, n=10):
    """rankings: list of lists of doc ids, each already sorted best-first."""
    score = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            score[doc_id] += 1.0 / (k + rank)
    return sorted(score, key=score.get, reverse=True)[:n]

fused = rrf([bm25_results, vector_results])

Work an example. Document A is ranked 1st by BM25 and 30th by the vector search: 1/61 + 1/90 = 0.0164 + 0.0111 = 0.0275. Document B is ranked 5th and 4th: 1/65 + 1/64 = 0.0154 + 0.0156 = 0.0310. B wins. That is the behaviour you want — a document both methods agree is good beats one that only a single method loves — and it falls out of the formula without any tuning.

The constant k controls how sharply rank 1 dominates. At k=60, the published default, the gap between rank 1 and rank 2 is small enough that agreement across methods can overturn it; at k=1 the top hit of either list would be nearly unbeatable. Leave it at 60 unless you have an evaluation set that says otherwise.

Making it work in practice

• Over-fetch each arm. Ask both retrievers for 50 and fuse down to 10. Fusion cannot rescue a document that neither list contained, and the cheapest way to raise recall is a longer candidate list.
• Run them concurrently. Latency is the max of the two, not the sum, and BM25 on a modest index is typically faster than the query embedding call it is racing.
• Keep the analyser honest. The lexical arm is only good at identifiers if your tokeniser preserves them. A default analyser that splits on punctuation turns ERR_TLS_CERT into three common words and throws away the exact thing you added BM25 for.
• Fuse, then rerank. RRF gives you a good candidate set cheaply; a cross-encoder orders it well. They are complementary stages, not alternatives, and the combination is the standard production shape.

One question RRF does not answer: what if you want to weight the arms unequally? Some stores expose a single alpha parameter blending normalised scores instead, and it is tempting because it is one dial. Resist tuning it without an evaluation set — alpha interacts with the score distributions of both retrievers, so a value tuned on one corpus transfers badly to another, and it will drift the moment you change embedding models. If you do need asymmetry, weight the RRF contributions instead: multiply each arm’s reciprocal term by a constant. That keeps the rank-only robustness and makes the weighting explicit.

It is also worth knowing when hybrid is not worth the second index. If your corpus has almost no proper nouns, identifiers or version numbers — narrative text, transcripts, generic prose — the lexical arm adds little beyond what the dense one already finds, and you are maintaining an inverted index for a few percent. Look at a sample of real queries first and count how many contain a token that must match exactly. If it is under one in twenty, spend the effort on reranking instead.