Testing That a Retriever Returns the Same Documents After a Reindex

A rebuild is supposed to change nothing. It nearly always changes something, and the useful question is not whether the results are identical — they will not be — but whether they moved more than a rebuild is entitled to move them.

Why the results legitimately move

Several parts of a vector index are not obliged to be deterministic, and a test asserting exact equality will fail on a rebuild where nothing is wrong.

• Approximate nearest-neighbour search. Graph indexes are built incrementally and depend on insertion order; two builds over the same vectors in a different order produce different graphs, and a query can traverse to a different local answer. That is what “approximate” means, and it is the setting you chose when you picked the index type.
• Ties. Two chunks with equal or near-equal scores can come back in either order unless the implementation breaks ties deterministically. Near-duplicate documents make this common.
• Floating-point summation order. Different batch sizes during embedding produce results that differ in the last bits, which is enough to swap two close neighbours.
• Corpus drift. If the rebuild also ingested new documents — as a production rebuild usually has — some movement is correct behaviour, not a regression.

Only an exact, brute-force index over an unchanged corpus with a deterministic tie-break gives you exact equality. If you have that, assert it. Most people do not.

Two measures that survive that

Two numbers per query, and they fail for different reasons, which is the reason to compute both.

Overlap at k. Of the top k results before and after, how many appear in both? This is a set comparison, immune to reordering, and it answers “is the retriever still finding the same material”. Four of five in common is 0.8.

Rank displacement. For the documents present in both, how far did each move? The maximum, and the mean, over the query set. This catches the case overlap cannot see: the same five documents returned in a completely reversed order, which is overlap 1.0 and a serious change, because whatever fits in the context budget is taken from the top.

export function overlapAtK(before: string[], after: string[], k: number) {
  const a = new Set(before.slice(0, k));
  const b = after.slice(0, k).filter((r) => a.has(r));
  return b.length / k;
}

export function displacement(before: string[], after: string[]) {
  const pos = new Map(after.map((r, i) => [r, i]));
  const moves = before
    .map((r, i) => (pos.has(r) ? Math.abs(pos.get(r)! - i) : null))
    .filter((n): n is number => n !== null);
  return {
    max: moves.length ? Math.max(...moves) : 0,
    mean: moves.length ? moves.reduce((s, n) => s + n, 0) / moves.length : 0,
  };
}

Compare on stable chunk identifiers, never on text or on position. Chunk text can be identical across two different documents, and a reindex that assigns new internal ids will report a total change if you compare on those. If your chunks have no stable identifier that survives a rebuild, that is the first thing to fix — a content hash of the chunk text plus its document path is enough, and it makes every test on this page possible.

Capturing a baseline and comparing

The query set should be a few dozen real queries, not synthetic ones: the head of your traffic, plus the awkward ones you know about. Capture the baseline from the live index before the rebuild, into a committed file.

import { describe, it, expect } from "vitest";
import baseline from "./fixtures/retrieval-baseline.json";

describe("reindex stability", () => {
  it("keeps the same documents in the top 5", async () => {
    const report: Record<string, number> = {};
    for (const [query, before] of Object.entries(baseline)) {
      const after = (await retriever.search(query, 5)).map((h) => h.ref);
      report[query] = overlapAtK(before as string[], after, 5);
    }
    const scores = Object.values(report);
    const mean = scores.reduce((s, n) => s + n, 0) / scores.length;
    const worst = Object.entries(report).sort((a, b) => a[1] - b[1]).slice(0, 5);

    console.table(worst);
    expect(mean).toBeGreaterThanOrEqual(0.9);
    expect(Math.min(...scores)).toBeGreaterThanOrEqual(0.6);
  });
});

A mean floor and a per-query floor together. The mean alone hides one query that collapsed among fifty that did not; the per-query floor alone is noisy on the tail. Printing the five worst before asserting means a failure names the queries to look at, which is the difference between a test that gets investigated and a test that gets its threshold lowered.

Choosing the tolerance

Pick the numbers empirically rather than from intuition, and the way to do that is honest: rebuild the index twice from the same corpus with no change at all, and measure the overlap between those two builds. That is your noise floor — the movement the system produces when nothing happened. Set the threshold below it with a margin. Any number chosen without that measurement is either so loose it never fires or so tight it fires every week until someone deletes the test.

Where the rebuild legitimately ingested new documents, compare against the subset of queries whose answering documents were already present, or accept a lower floor and say so in the test name. Silently loosening a threshold to accommodate corpus growth turns the test into decoration.

Two smaller choices make the band easier to live with. Compare at the k production uses rather than a larger one — overlap measured at 20 when you pass 5 chunks to the model reports stability in a region nobody reads. And weight the query set towards traffic: fifty queries drawn evenly from a long tail will report a worse and less meaningful number than fifty drawn from what people actually ask, because tail queries have weaker score separation and therefore move more for entirely innocent reasons.

When a difference is not drift

Some failures have a shape that identifies them immediately, and it is worth recognising them before reaching for the tolerance.

• Overlap near zero everywhere. Not drift. Either the embedder changed — a provider updating a model behind the same name does this — or the index is being queried with vectors from a different model than it was built with. Dimension mismatches usually error; a same-dimension model change does not.
• Overlap fine, displacement large and one-directional. Ordering is inverted somewhere: a distance being treated as a similarity, or a sort direction flipped. Cosine distance and cosine similarity are opposites and are easy to swap during a migration.
• A subset of queries at zero, the rest perfect. Documents are missing. A partial ingest, a failed batch, a filter excluding a source. Check the document count before investigating ranking.
• Everything shifted by one position. A duplicate document is now being returned above the rest. Look for a re-ingested source rather than a ranking change.

Run this comparison against the new index before it takes traffic, not after. A rebuild is one of the few retrieval changes that can be staged, compared and rolled back cleanly, and the comparison is worth nothing if it happens after the switch.