Keyword vs Semantic Search: The Same Query, Both Rankings

Runs BM25 and a vector ranking over the same documents side by side, with reciprocal rank fusion, and counts the query terms that score exactly zero on the lexical side.

Query — the one control worth movingSwap a word for a synonym the documents never use, and watch the two columns come apart.

Documents — one per lineEight illustrative support snippets. Replace them with your own corpus; both methods are computed from whatever is in this box.

BM25 k₁term-frequency saturation; 1.2 is the Lucene default

1.20

BM25 blength normalisation; 0.75 is the Lucene default

0.75

Vector sideWhere the second ranking comes from.

LSA rankHow many latent dimensions the documents are projected onto before cosine. Low rank merges vocabulary; high rank approaches plain TF-IDF.

rank 3

Query terms that appear in no document at all

6 of 7

BM25 scores “how”, “do”, “i”, “change”, “credential”, “safely” at exactly zero — not low, zero. Those terms cannot contribute to a lexical ranking however important they are, which is the entire case for a second retriever.

Hybrid	BM25	Vector	Document
#1	#1 (0.98)	#1 (1.00)	Rate limits are applied per key, so a freshly created key starts with a clean limit window.
#2	#2 (0.91)	#2 (0.98)	Credentials should be replaced on a schedule. A superseded secret stays valid throughout the overlap window so callers never see an error.
#3	#3 (0.71)	#3 (0.55)	Invoices are issued monthly and can be downloaded as a PDF from the billing page at any time.
#4	#4 (0.69)	#6 (0.33)	If a secret has leaked, revoke it immediately and accept the downtime. Graceful replacement is for planned changes only.
#5	#7 (0.00)	#4 (0.40)	Webhook signing secrets are stored separately from API keys and are cycled on their own schedule.the two methods disagree by 3 places
#6	#5 (0.00)	#7 (0.27)	Rotating an API key without downtime: create the new key, deploy it everywhere, confirm traffic has moved, then revoke the old one.
#7	#8 (0.00)	#5 (0.34)	The keys page lists every key you have created, with the date it was last used and its final four characters.the two methods disagree by 3 places
#8	#6 (0.00)	#8 (-0.28)	Refunds are returned to the original payment method and take five to ten working days to appear.

Documents: 8
Average document length: 19.0 words
Distinct query terms: 7
Query terms with zero document frequency: 6
Vector ranking produced by: latent semantic analysis at rank 3
BM25 puts first: Rate limits are applied per key, so a freshly create
The vector side puts first: Rate limits are applied per key, so a freshly create
Documents the two rank ≥3 places apart: 2
Fusion: reciprocal rank fusion, k = 60

Where these numbers come from: The BM25 column is BM25 — the Lucene formula, computed on the documents in the box. The vector column, unless you pasted your own embeddings, is latent semantic analysis: a TF-IDF matrix truncated to a few dimensions, which is a real vector-space retrieval method but one built from these documents only. It is not a modern embedding model and it will not behave like one — it can only relate words that co-occur in the corpus in front of it. That limitation is stated here rather than hidden behind a plausible-looking number, and the mode switch above lets you replace it with the real thing.

What this assumes: BM25 with idf = ln(1 + (N − df + 0.5)/(df + 0.5)), the variant Lucene and Elasticsearch use, so a term in every document contributes near zero rather than going negative. Tokens are lowercase alphanumeric words: no stemming, no stopword list, no lemmatisation — a real lexical index has all three and would close some of the gaps you see here. LSA truncates TF-IDF by power iteration and takes cosine in the reduced space; at rank equal to the number of documents it converges on plain TF-IDF cosine. Hybrid uses reciprocal rank fusion at k = 60, which combines ranks rather than scores precisely because BM25 scores and cosine similarities are not on a comparable scale.

Why you end up running both

The two methods fail in ways that do not overlap, and that is the whole argument for hybrid search. BM25 matches terms. It is fast, needs no training, handles a corpus it has never seen, and gives you an explanation for free — the score decomposes into a contribution per query term. Its failure is total rather than graceful: a query term that appears in no document contributes exactly zero, so a user who asks about "credentials" when your documentation says "API keys" gets nothing at all from the lexical side.

Vector retrieval fails the other way. It has no notion of a term appearing or not appearing, so it always returns a full ranking — which is useful when the vocabulary differs and dangerous when there is nothing relevant to return, because the top result looks the same either way. And it is unreliable on the things BM25 is best at: product codes, error numbers, surnames, version strings. An embedding of ERR_4013 is close to the embedding of ERR_4031, which is exactly the wrong behaviour.

Fusion is how production systems get both. Reciprocal rank fusion adds 1/(60 + rank) from each ranker, deliberately ignoring the scores: a document needs to place well in one list to survive, and placing well in both wins. The number worth watching above is the count of documents the two methods rank three or more places apart. When it is high, hybrid is buying you something real. When it is near zero, your queries and your corpus share a vocabulary and the second retriever is latency you are paying for nothing.

Keyword vs Semantic Search: The Same Query, Both Rankings

Why you end up running both

The reasoning behind this

Related tools