What Changes in Semantic Search Quality After a Migration

“Search feels worse since the migration” is not yet a bug report. Turning it into one takes a fixed query set, one comparison, and one plot that tells you whether you are looking at a model or at a job that stopped halfway.

Freeze a query set before you touch anything

The single most valuable artefact in a retrieval migration costs an afternoon and is almost never made: a few hundred real queries, drawn from your logs, with the identifiers of what the current system returns for each. Not judgements, not labels — just the queries and the current top-k identifier lists, captured before the change.

Sample deliberately rather than taking the head. Include the most frequent queries because they dominate perceived quality, but also a stratified sample across query length, language, and whether the query uses a filter — those are where a migration breaks and they are absent from a head sample. Add every query that has ever appeared in a bug report. Two hundred queries is enough to see a real effect; two thousand is better and costs nothing extra to run.

If you are reading this after the deploy and did not capture a baseline, you can still recover one if the old index still exists — which is the argument for the dual-running described in migrating a RAG pipeline between clients. If it does not, you are reduced to human judgement on the new results, which is slower and much less conclusive.

Overlap at k, and reading the number

The primary measure needs no labels: for each query, what fraction of the old top-k identifiers appear in the new top-k. Report the distribution, not the mean — the mean hides the shape, and the shape is the diagnosis.

def overlap_at_k(old, new, k=10):
    a, b = set(old[:k]), set(new[:k])
    return len(a & b) / float(k)

# report the distribution, not the average
rows = [(q, overlap_at_k(before[q], after[q])) for q in queries]
buckets = collections.Counter(round(o, 1) for _, o in rows)

What the shape tells you:

• A broad spread centred somewhere in the middle is what an intentional model change looks like. Different model, different space, different neighbours — the ordering genuinely changed and some of the change is probably an improvement. This is not a regression signal on its own.
• A bimodal distribution — most queries near 1.0, a distinct group near 0 — means something is different for a subset rather than for everything. That is the signature of a partial problem: an unfinished re-index, one namespace missed, one language or document type handled differently.
• Near 1.0 everywhere after a model change means the new model is not being used on the query side. Check the query path separately from the index path; they are frequently configured in different places.

Overlap tells you what moved, not whether it got worse. For that, judge only the differences: take the identifiers that dropped out and the ones that came in, and have somebody rate those. That is a fraction of the work of rating everything and it is the only part that carries information.

Model or index: the discriminating test

Two causes produce the same complaint and have opposite fixes. Either the new model genuinely ranks your corpus differently, or part of the corpus is still in the old vector space and is competing badly against the part that is not.

The test that separates them: run the query set, collect every returned score together with the stored model tag on the record that produced it, and histogram the scores grouped by tag. Two populations means a mixed index. As described in what happens to an embedding cache on a model change, similarity computed between vectors from two unrelated training runs lands in a tight band near zero with no relationship to meaning, so old-space records show up as a narrow spike well below the new-space spread. A genuine model effect shows one population.

Two corroborating checks, both quick. Count records by model tag per namespace, partition or shard — a job that died usually died in one partition, and the affected partition is the one whose counts do not match. And look at the documents that dropped out of results: if their creation timestamps or ingestion batches cluster, you are looking at a job boundary rather than a semantic effect. If they cluster by topic or language instead, the model is the story.

If the tags are not there, add them before doing anything else, because without them this test does not exist and you will be arguing from anecdote. Auditing whether a re-embedding actually finished builds the check that would have caught this before the deploy.

The query-side asymmetry people drop

Assume the model is the cause and the index is complete. Before concluding the model is worse, check that you are calling it the way it expects to be called.

Many retrieval embedding models are asymmetric: they are trained so that a query and a passage are embedded differently, and the difference is expressed either as a required text prefix or as an API parameter distinguishing a query from a document. Two families of mistake follow a migration. Moving from a symmetric model to an asymmetric one and not adding the distinction — every query is embedded as if it were a document, which degrades ranking across the board without failing. Moving the other way and leaving the old prefixes in place, so the literal prefix text becomes part of the content being embedded.

Both are cheap to test: embed one query with and without the distinction, retrieve with each, and compare. If the results differ substantially, the parameter matters for this model and you need to get it right; if they are nearly identical, rule it out and move on. Read the model card for the specific model rather than assuming the convention carries across a family — the requirement is a property of how that model was trained, and it is stated there.

Metric, normalisation and truncated dimensions

Three more mechanical causes, each of which produces a quality drop without an error.

• The distance metric changed with the store. If the old index used cosine and the new collection was created with inner product or L2, ranking changes for any vector set that is not unit-length — inner product rewards long vectors, cosine ignores length entirely. Check the collection’s configured metric against the old one rather than assuming the default matched.
• Normalisation moved. Some models return unit-length vectors, some do not, and some pipelines normalise on the way in. If the old pipeline normalised and the new one does not, an inner-product index silently starts ranking by length. Compute the L2 norms of a sample from both sides and compare the distributions.
• The output was shortened. Where an API lets you request fewer dimensions than the model natively produces, the shorter vector is a genuine trade of retrieval quality for storage and speed. If somebody chose a smaller width during the migration to save on storage, that is a deliberate quality decision that may have been made without anyone framing it as one. Embedding dimensions covers what the trade actually costs.

Also re-tune any similarity threshold. A cut-off calibrated against one model’s score distribution has no reason to hold against another’s, and a threshold that was filtering 5% of results before may be filtering 40% after. Plot the score distribution on the new model and pick the cut-off from it rather than carrying the constant over.

Deciding to roll back

Write the rollback trigger down before the cutover, as a number and a window, because after the cutover everyone is invested. Something of the form: median overlap below a stated floor, or the share of queries with overlap under 0.3 above a stated ceiling, measured on the frozen query set, at a stated point after the flip. Those are computable without human labelling, which is what makes them usable as a trigger rather than as a discussion topic.

Keep the old index queryable until that window closes. The cost of storing a duplicate index for two weeks is small and knowable; the cost of discovering a regression with nothing to fall back to is a re-embed of the entire corpus under time pressure. Roll back with a flag, then diagnose with the plots above at your own pace.