Testing That Cosine Similarity Thresholds Don't Drift After a Model Update

Retrieval starts returning nothing, or starts returning everything. Nobody changed the threshold, the query code, or the corpus. What changed is the embedding model, and the number 0.82 in your filter no longer refers to the same thing.

The symptom

It arrives in one of two shapes and they are the same bug. Either the “no relevant documents found” path starts firing on queries that obviously have answers, or the opposite: every query returns the maximum number of results and the answers become vague because the context is full of near-misses. Both appear immediately after a re-index, a provider migration, a dimension change, or a silent model update — and often after a change nobody connected to retrieval at all.

The reason it is confusing is that the retrieval order is usually fine. The most similar document is still the most similar document. Only the absolute scores moved, and the only thing that reads absolute scores is your cut-off.

A threshold is a cut through a distribution

Cosine similarity is not calibrated. Nothing about a score of 0.82 means “82% relevant”; it is a position in a distribution that a particular model happens to produce over a particular corpus. Different models place that distribution differently. Some spread scores across most of the range; others compress everything unrelated into a narrow band near 0.7, so that a threshold below that band matches every document in the index.

So the meaningful quantity was never the score. It was the separation between the scores of relevant pairs and irrelevant ones, and the threshold was a cut somewhere in the gap. When the model changes, the gap moves and often changes width. A constant cannot follow it, which means a hardcoded threshold is a bug with a delayed fuse in every system that has one.

The worse version: a mixed index

There is a failure adjacent to this one that is worth ruling out first, because it is more damaging and easier to cause. If a re-index was partial — some vectors from the old model, some from the new — then the index contains two incomparable coordinate systems, and similarity between a new query and an old vector is meaningless rather than merely shifted. Symptoms are erratic instead of uniformly wrong: some queries fine, some nonsensical, no pattern by topic.

The guard is structural. Store the model id and dimension alongside every vector, and refuse at query time to compare vectors whose model id differs from the query’s. A test for it is three lines and it is worth having permanently, because the situation recurs every time a re-index is interrupted.

it("refuses to score across embedding models", async () => {
  await index.upsert({ key: "doc-a", vector: v1, model: "text-embedding-3-small" });
  await expect(
    index.search({ vector: v2, model: "text-embedding-3-large" }),
  ).rejects.toThrow(/model mismatch/);
});

The test that catches it

You need a small labelled set: pairs of query and document marked relevant, and pairs marked irrelevant. Fifty of each is enough to be useful, and they should come from real queries — the ones from your logs that people actually type, including the short ambiguous ones.

The assertion is not about the threshold. It is about separation: the worst relevant pair should still score above the best irrelevant pair, or if they overlap, the overlap should be no worse than it was. That is a property of the model and the corpus, and it either survives a model change or it does not.

// threshold-drift.test.ts
import { expect, it } from "vitest";
import { embed, cosine } from "../src/embeddings";
import { RELEVANT, IRRELEVANT } from "./fixtures/labelled-pairs";

async function scores(pairs: Array<[string, string]>) {
  const out: number[] = [];
  for (const [q, d] of pairs) {
    const [qv, dv] = await embed([q, d]);
    out.push(cosine(qv, dv));
  }
  return out.sort((a, b) => a - b);
}

it("keeps relevant and irrelevant pairs separated", async () => {
  const rel = await scores(RELEVANT);
  const irr = await scores(IRRELEVANT);

  const p5Relevant = rel[Math.floor(rel.length * 0.05)];
  const p95Irrelevant = irr[Math.floor(irr.length * 0.95)];

  // The gap, not the absolute numbers, is what must survive a model change.
  expect(
    p5Relevant - p95Irrelevant,
    "5th pct relevant " + p5Relevant + " vs 95th pct irrelevant " + p95Irrelevant,
  ).toBeGreaterThan(0.05);
}, 120_000);

Percentiles rather than min and max, deliberately. A single mislabelled pair in the fixture makes a min-versus-max assertion fail forever and teaches everyone to ignore it; the fifth and ninety-fifth percentiles tolerate a couple of bad labels while still failing when the distributions genuinely collide. Print both numbers in the failure message, because the interesting information is which side moved.

Deriving the threshold instead of writing it down

The fix that ends the problem is to stop having a constant. Compute the threshold from the labelled set as part of your build or your re-index, write it to a generated file, and have the application read it. A reasonable rule is the value that maximises whichever of precision or recall you care about on the labelled set, or more simply a chosen percentile of the irrelevant distribution.

1. Re-embed the labelled set with the model you are about to deploy.
2. Compute the two score distributions and pick the cut — for example, the 98th percentile of the irrelevant scores, so roughly two per cent of irrelevant pairs get through.
3. Write it to a generated file with the model id and the date beside it, and fail the build if the resulting recall on the relevant set falls below what the previous model achieved.
4. Deploy the model and the threshold together, in one change. A model rollout that does not carry its threshold is the original bug wearing a different hat.

This also gives you something to point at when a provider updates a model without telling you: the same job, run on a schedule against an unchanged labelled set, reports a moving threshold as a signal. A drifting cut-off is one of the earliest observable consequences of weights changing underneath you, and it is considerably cheaper to watch than a full evaluation.

One caveat about the labelled set, since everything above rests on it. It is a fixture and it rots: queries that were representative a year ago are not, and a set assembled from the first week of logs over-represents whatever people were doing that week. Add to it from real failures — when retrieval visibly misses, that query and the document it should have found become a labelled pair — and resist removing pairs because they fail. A pair that consistently fails is either a labelling mistake, which you should fix, or a genuine weakness, which you should keep.

Finally, be clear about what this test does not tell you. Separation on a labelled set is a property of retrieval, not of the answers your system produces; a system can retrieve the right document and still answer badly, and the reverse. Keep the threshold test as the fast, cheap, deterministic check that runs on every re-index, and let a full golden dataset carry the end-to-end question. They fail for different reasons, and a suite that conflates them cannot tell you whether the retrieval or the generation moved.