Semantic Search vs Keyword Search: Where Each Fails

The framing that semantic search superseded keyword search is commercially convenient and empirically shaky. BEIR (Thakur et al., 2021) was built specifically to test dense retrievers out of domain, and its headline finding was that BM25 remains a strong baseline many of them fail to beat. Knowing exactly which queries those are is the useful part.

Two mechanisms, not two qualities

BM25 scores a document by the query terms it contains, weighting each by how rare the term is in the collection and damping the contribution of repeats:

score(q, D) = sum over terms t in q of

    idf(t) *  f(t,D) * (k1 + 1)
             --------------------------------------
             f(t,D) + k1 * (1 - b + b * |D| / avgdl)

  idf(t) is large for rare terms, near zero for common ones
  k1 ~ 1.2 damps term repetition;  b ~ 0.75 normalises document length

Two properties of that formula drive everything below. It is exact — a term either appears or it does not, with no notion of nearby meaning. And it is sharply weighted by rarity, so a single unusual token in the query dominates the score. A dense retriever has neither property: it compares one summary of the query against one summary of the document, and a rare token is one of several hundred contributions to a vector that was already compressed.

Six archetypes where BM25 wins

• Identifiers and codes. ERR_MODULE_NOT_FOUND, CVE-2024-3094, an order number, a SKU. BM25 gives these a huge idf and finds the one document containing them. A dense model has no representation for an arbitrary string and will return documents about the general topic instead — often plausibly, which is worse than returning nothing.
• Rare proper nouns. Your internal project name, a customer’s company, a specific library version. Rare in the collection means high idf for BM25 and out-of-vocabulary for the embedding model.
• Exact-phrase intent. Someone pasting an error string wants the document containing that string, not documents about similar errors.
• Negation. “deployments that did not fail” embeds within a hair of “failed deployments”. BM25 does not understand the negation either, but it at least does not confidently rank the opposite first.
• Domain jargon the model never saw. A model trained on general web text has weak representations for specialist vocabulary — pharmaceutical compound names, legal terms of art, proprietary part numbers. Rarity is exactly what BM25 rewards and what a general embedding model handles worst.
• Very short queries in a narrow corpus. A two-word query gives the embedding model almost nothing to work with, while the corpus being narrow means every document is semantically close to every other and the vector scores compress into a band where the ordering is noise.

Where dense wins outright

The complementary list is short but it covers the queries people actually type into a product. Vocabulary mismatch: “how do I stop being billed” against a page titled “Cancelling your subscription”, where the two share no content word and BM25 scores zero. Descriptive queries: “the thing that turns a PDF into text”, where the user does not know the term and therefore cannot type it. Cross-lingual retrieval, where a multilingual model places a Dutch query near a German document and no lexical index ever will. And tolerance of typos and morphology, where subword tokenisation degrades gracefully instead of missing.

Notice these are the queries in a support search box and those are the queries in a developer’s console. Which retriever is better is genuinely a question about who is typing.

Fusing them, with the arithmetic

You do not have to choose, and the standard way not to choose is reciprocal rank fusion (Cormack, Clarke and Buettcher, SIGIR 2009). It combines ranked lists using only the ranks, never the scores:

rrf(d) = sum over retrievers r of  1 / (k + rank_r(d)),   k = 60

  doc     bm25 rank   dense rank   rrf
  A            1           -        1/61            = 0.01639
  B            -           1        1/61            = 0.01639
  C            3           2        1/63 + 1/62     = 0.03200
  D            2           8        1/62 + 1/68     = 0.03083

  fused order: C, D, A, B

The worked example shows why it behaves well. C is second-best in neither list and finishes first, because appearing respectably in both lists is stronger evidence than topping one. The constant k = 60 is the value from the original paper; it flattens the difference between rank 1 and rank 5 so that a single retriever cannot dominate on confidence alone.

The reason to prefer this over a weighted sum of scores is that BM25 scores and cosine similarities live on incomparable scales, and the scale of BM25 depends on the collection and the query. Normalising them into agreement requires calibration that goes stale as the corpus changes. Ranks need no calibration at all, which is why RRF has survived as the default despite being the simplest thing in the field.

What to build first

One implementation note that removes most of the objection to running two retrievers: if your data is in Postgres you already have both. A tsvector column with a GIN index gives you lexical ranking, a vector column with HNSW gives you the dense side, and the fusion is a CTE per retriever joined on id with the reciprocal-rank expression in the select list. One query, one transaction, no consistency problem between two stores and nothing extra to operate. Postgres’s own ranking function is not BM25 by default, which is worth knowing, but the fusion step cares only about ranks.

Build the lexical index first. It is cheap, it has no model dependency, it never needs re-embedding, and it will handle the identifier queries that a dense-only system fails on most visibly. Add the dense retriever second and fuse. Then instrument the fused system so you can see, per query, which retriever contributed the result that was clicked — that log is what tells you whether the second system is earning its operating cost, and it is the artefact almost nobody builds.

If you can only run one, ask what your users type. Support search and product discovery skew dense. Documentation search, log search and anything a developer touches skews lexical far more than the current discourse suggests.