Snapshot Testing LLM Output With Jest

Jest snapshots are a good tool pointed at the wrong value. Pass a raw completion to toMatchSnapshot() and you have pinned the sampler’s exact draw; the fix is to snapshot a reduction of the response, and a custom serializer is where that reduction belongs.

Why a raw snapshot of a completion is useless

Write expect(res.choices[0].message.content).toMatchSnapshot() and the first run writes several hundred words into __snapshots__/answer.test.ts.snap. The second run produces prose that means the same thing with a different second sentence, and Jest prints a diff in which every line after the divergence is marked changed. Nothing about that diff tells you whether the model got the refund window wrong. It tells you that the text is different, which you already knew before you wrote the test.

The reviewer’s response to that diff is the real damage. Faced with a wall of red and green that is usually meaningless, people press -u. Once that reflex exists the suite no longer detects anything, because the day the answer is genuinely wrong the diff looks exactly like the previous forty. See why exact-match snapshots fail for the sampling mechanism underneath this.

Decide what the snapshot is a snapshot of

Before any Jest code, write down the fields whose change you would want to be told about. For a support-answer endpoint that is usually something like: the model id actually served, the finish reason, the names of any tools called and in what order, whether a refusal was produced, and a small set of extracted facts — a currency amount, a number of days, a policy identifier. Wording is not on that list. Token count is not on that list.

Build that object in a plain function next to the test, tagged so a serializer can recognise it. Everything volatile — request ids, latency, timestamps — either does not go in or gets handled in the next section.

The test for whether the list is right is simple: for each field, name the failure it would catch. finishReason catches truncation against your max_tokens, which produces answers that look fine until the last sentence stops mid-clause. The tool-call array catches a prompt edit that stopped the model reaching for the lookup at all, which is the most common regression in a tool-using app and one that free-text assertions almost never see. refused catches a safety filter engaging on an input that used to pass. If a field has no such answer, it is in the snapshot because it was in the response, and it will generate diffs nobody can act on.

// tests/support/summarise.ts
export type LlmAnswer = {
  __type: "llmAnswer";
  model: string;
  finishReason: string;
  toolCalls: string[];
  refused: boolean;
  facts: Record<string, string | null>;
};

const DAYS = /\b(\d{1,3})\s+days?\b/i;

export function summarise(res: any): LlmAnswer {
  const msg = res.choices[0].message;
  return {
    __type: "llmAnswer",
    model: res.model,
    finishReason: res.choices[0].finish_reason,
    toolCalls: (msg.tool_calls ?? []).map((t: any) => t.function.name),
    refused: msg.refusal != null,
    facts: {
      refundWindowDays: msg.content?.match(DAYS)?.[1] ?? null,
    },
  };
}

The regex is doing real work and it is the part to be honest about: it is a fact extractor, and if the model expresses the same fact as “a month” it returns null and the snapshot records null. That is a visible, reviewable failure rather than a silent one, which is the property you want. A more robust extractor is a schema — see snapshotting structured JSON output.

The serializer

Jest formats snapshot values with pretty-format, and a custom serializer is a pretty-format plugin: an object with a test(val) predicate and a serialize function. The modern plugin signature is serialize(val, config, indentation, depth, refs, printer); the older print(val, printer, indenter, config, colors) form still loads but does not get told the depth or the refs, so recursion through nested values behaves worse. Use serialize.

// tests/serializers/llm-answer.js
const KEY_ORDER = ["model", "finishReason", "toolCalls", "refused", "facts"];

module.exports = {
  test: (val) =>
    val != null && typeof val === "object" && val.__type === "llmAnswer",

  serialize(val, config, indentation, depth, refs, printer) {
    const ordered = {};
    for (const key of KEY_ORDER) {
      if (val[key] !== undefined) ordered[key] = val[key];
    }
    return "LLMAnswer " + printer(ordered, config, indentation, depth, refs);
  },
};

Two things are happening. The test predicate keeps the plugin off every other value in your suite, which matters because a serializer registered globally sees everything Jest ever prints. And the explicit KEY_ORDER loop is not cosmetic: it is the only place you can fix key ordering. Jest’s snapshotFormat config accepts pretty-format options, but its documentation names compareKeys and plugins as the two it will not take, so you cannot sort keys from configuration. Rebuild the object in a fixed order and the problem disappears.

Register it globally rather than in each file, so a new test cannot accidentally snapshot the raw shape:

// jest.config.js
module.exports = {
  snapshotSerializers: ["<rootDir>/tests/serializers/llm-answer.js"],
  snapshotFormat: { printBasicPrototype: false },
};

expect.addSnapshotSerializer(serializer) does the same thing for a single test file when you want the plugin scoped narrowly. Jest applies the most recently added plugin first.

Property matchers for the fields you cannot drop

Some volatile fields have to stay because their presence is the point. A response id proves a call was made; you do not want to assert its value. toMatchSnapshot(propertyMatchers, hint) takes an object of asymmetric matchers, checks those properties like toMatchObject, and writes the matcher’s own printed form into the stored snapshot instead of the value.

test("refund policy answer", async () => {
  const res = await ask("How long do I have to return a jacket?");
  expect({ ...summarise(res), requestId: res.id, latencyMs: res.latency })
    .toMatchSnapshot(
      { requestId: expect.any(String), latencyMs: expect.any(Number) },
      "refund window",
    );
});

The stored file records Any<String> where the id was, so the test fails if the field vanishes or changes type and passes when it merely changes value. The second argument is the hint: a string appended to the snapshot name, which is what lets one test hold several snapshots without them being told apart only by an index.

Stopping CI from writing snapshots

A missing snapshot is written silently on first run. On a build agent that is a test that cannot fail, because the first thing it does is record whatever it saw as correct. Run Jest with --ci in the pipeline: new snapshots are then reported as failures instead of being written, and the only way a snapshot enters the repo is a person running -u locally and committing the result.

There is a second reason to care about --ci here specifically. Every one of these tests costs a real API call, so a CI run that silently writes snapshots is also spending money to record a result nobody will ever compare against. Whatever you do about network access in tests — a recorded cassette, a stub, a live call on a nightly job only — decide it deliberately rather than discovering it from a bill. Testing without the model covers when a live call is worth its cost and when a recording is enough.

The matching hazard is obsolete snapshots. Deleting a test leaves its entry in the .snap file forever; Jest reports the count, and --ci will not clean it. Removing them is part of the regeneration workflow, not something to do by hand at random.