Snapshot Testing Retrieved Chunks Before They Reach the Prompt

By the time a retrieval regression reaches the answer, it has been rewritten into fluent prose by a model that will happily explain whatever it was given. Catching it one layer earlier — as a diff in which chunks came back — turns an unattributable quality complaint into a line somebody can read.

Why snapshot here and not the answer

Snapshotting a generated answer is the canonical mistake. The output is non-deterministic, so the snapshot fails on runs where nothing changed; people respond by updating it; and after a few rounds the file records whatever the model said most recently rather than anything anyone decided. It has become a log with a test framework attached.

Retrieved chunks are different in the way that matters. For a fixed index and a fixed query, retrieval is deterministic or very nearly so, so a changed snapshot means something changed in the system. And the content is short, structured and human-checkable: a reviewer can look at five chunk ids and titles and say whether that is a better or worse set for the question, which is not true of two paragraphs of prose.

It also sits in the right place for attribution. Between a retriever tested against a fake corpus and generation tested against fixed context, this is the one check that runs against your real index and real documents, and it is the seam where those two halves meet.

Snapshot a projection, not the objects

Never snapshot the retrieval result as returned. It contains vectors, floating-point scores, internal ids and timestamps — all of which churn without meaning anything, and any one of which makes the diff unreadable and the snapshot untrusted.

type Hit = { ref: string; doc: string; text: string; score: number };

export function projection(hits: Hit[]): string {
  return hits
    .map((h, i) => {
      const firstLine = h.text.replace(/\s+/g, " ").slice(0, 80);
      return `${String(i + 1).padStart(2)}. ${h.doc}#${h.ref}\n    ${firstLine}`;
    })
    .join("\n");
}

Rank, source document, chunk id, and eighty characters of normalised text. That is enough for a reviewer to recognise the chunk and for the diff to show a reordering, a substitution or a boundary shift as a small readable change.

Scores are deliberately absent, and this is the decision the whole technique rests on. Including them means every embedding update, every index parameter tweak and every floating-point difference rewrites the file, so the snapshot fails constantly for reasons nobody acts on — and a test that fails for reasons nobody acts on gets updated without being read, at which point it stops working entirely. If score movement is what you want to watch, measure it as a number with a tolerance, which is what the reindex comparison does. Truncating the text to a fixed length has the same motivation: an edit deep inside a document should not rewrite a snapshot of what was retrieved.

File snapshots, one per query

Inline snapshots put the expected value in the test file, which is convenient for short values and wrong here — five chunks is twenty lines, and twenty queries makes the test file unreadable. Vitest supports snapshotting to an explicit path with toMatchFileSnapshot, documented in its expect API reference, and that gives you one reviewable file per query.

import { describe, it, expect } from "vitest";
import { projection } from "./projection";

const queries = [
  ["refund-timing", "how long does a refund take"],
  ["damaged-photo", "do I need a photo for a damaged item"],
  ["late-return-fee", "what is the fee for a late return"],
];

describe("retrieved chunks", () => {
  it.each(queries)("%s", async (name, query) => {
    const hits = await retriever.search(query, 5);
    await expect(projection(hits))
      .toMatchFileSnapshot(`./__snapshots__/retrieval-${name}.txt`);
  });
});

toMatchFileSnapshot returns a promise, so it must be awaited; without the await the assertion resolves after the test has finished and failures are reported against the wrong test or not at all. Naming the file after the query rather than the test index means reordering the list does not renumber every snapshot.

The update is the test

A snapshot suite is only as good as what happens when one changes, and the failure mode is entirely social. Updating snapshots in bulk to make a build green destroys the entire value of the technique, because the new file is then a record of a bug rather than of a decision.

1. Read the diff before regenerating anything. A chunk dropping out of the top five is a real event with a cause.
2. Decide which change you made could have caused it — the chunker, the embedder, the index, the corpus, the query preprocessing. If none of them could have, that is the finding, and it usually means something was ingested or removed that you did not know about.
3. Regenerate one query at a time, and commit the updated snapshots in the same change as the code that moved them, never separately.
4. Never run the update flag across the whole suite in a hurry. If that is tempting, the projection is too churny — fix the projection.

Keep the query list small, around ten to twenty. Small enough that somebody genuinely reads every diff, which is the property doing all the work.

Where snapshots stop being the right tool

A snapshot asserts that today matches yesterday. It has no opinion about whether either was correct, so a suite of snapshots taken while retrieval was already broken passes forever and encodes the bug. Pair it with at least a few assertions that state what is right: the required-span tests from the chunking regression suite are the natural partner, because they fail on the day retrieval breaks rather than on the day it changes.

Snapshots are also a poor fit where retrieval is genuinely non-deterministic — an approximate index with a low search-effort parameter, or a reranker sampled at non-zero temperature. Both make the file change without a cause, which is the same trap as snapshotting prose. Raise the search effort for the test run if that is available, and if it is not, measure overlap with a tolerance instead. And if your corpus changes hourly, snapshots against the live index will never be stable; run them against a pinned copy of the corpus, or accept that this check belongs in a staging environment rather than in CI.