Snapshot Testing Structured JSON Output From a Model

Structured output is the one case where a snapshot test earns its keep: a JSON object diffs on keys and types rather than on wording. It only works if you validate and normalise first, because an unnormalised object reintroduces exactly the noise you switched formats to escape.

Two assertions, not one

A structured response deserves two separate checks that fail for different reasons, and collapsing them into one snapshot loses the distinction.

The first is schema validity, asserted on every case with no golden file involved. Does the response parse as JSON, and does it satisfy the schema you asked for — required fields present, enums inside their allowed set, numbers inside their bounds? This assertion is deterministic, cheap, and the one that catches the failures that break your client code. Use the validator you already have: ajv against a JSON Schema, a zod schema’s safeParse, or a Pydantic model’s model_validate. Do not hand-roll it.

The second is the snapshot, which records the values. It answers a different question — did this input start producing a different label, a different bucket, a different set of extracted entities — and it is the one that needs a human when it fails. Keeping them apart means the failure message already tells the reader which kind of problem they have.

Normalising before the snapshot

Four things make a structurally identical object produce a diff, and all four are worth removing before the value reaches the snapshot.

• Key order. Serialisers preserve insertion order, and a model does not emit keys in a fixed order. In Jest this cannot be fixed from configuration: the documentation names compareKeys and plugins as the two pretty-format options snapshotFormat will not accept. Sort the keys in your own code.
• Array order, where the array is a set. A list of extracted entities usually has no meaningful order; sort it by a stable key. Where the order is meaningful, do not sort it, and say so in a comment so the next person does not.
• Float precision. A confidence of 0.8231 and 0.8234 are the same answer. Round, or bucket, and prefer bucketing where your code branches on a threshold anyway.
• Generated identifiers and timestamps. Drop them, or hold them with a type matcher so their disappearance is still caught.

The test

import { describe, expect, test } from "vitest";
import { z } from "zod";

import { extractEntities } from "../src/extract";

const Entity = z.object({
  type: z.enum(["person", "org", "location"]),
  text: z.string().min(1),
  confidence: z.number().min(0).max(1),
});
const Result = z.object({ entities: z.array(Entity) });

function normalise(raw: unknown) {
  const parsed = Result.parse(raw);
  return {
    entities: parsed.entities
      .map((e) => ({
        type: e.type,
        text: e.text,
        confidence: Math.round(e.confidence * 10) / 10,
      }))
      .sort((a, b) => a.text.localeCompare(b.text) || a.type.localeCompare(b.type)),
  };
}

describe("entity extraction", () => {
  test("press release with three orgs", async () => {
    const raw = await extractEntities(fixture("press-release-2024-11.txt"));

    // 1. Schema: deterministic, no golden file.
    expect(() => Result.parse(raw)).not.toThrow();

    // 2. Values: golden file, reviewed by a human when it moves.
    expect(normalise(raw)).toMatchSnapshot();
  });
});

normalise calls Result.parse rather than trusting its input, so the snapshot can never record a shape the schema would have rejected. Vitest also offers toMatchFileSnapshot, which writes to a path you choose — worth using here, because a .json extension gives you syntax highlighting and one file per case in review rather than one large .snap holding everything.

Notice which assertion carries the golden file. The schema check runs on every case and needs no recorded artefact, so it can be applied to a thousand inputs cheaply; the snapshot is the expensive one, because each stored file is a claim a person is eventually responsible for. In a large suite that asymmetry decides the design: validate everywhere, snapshot a chosen subset, and be able to say why each case is in the subset.

Two details in the normaliser are worth copying rather than reinventing. The sort uses a tie-breaker, because a comparison on text alone leaves two entities with the same surface form in an unspecified order and reintroduces a diff that means nothing. And the rounding is done on the way into the snapshot rather than in the assertion, so the stored file contains the rounded value — a reviewer reading the golden file sees exactly the number the test compares, which is not true if the tolerance lives in the matcher.

The same test in Python

from pydantic import BaseModel, Field
from syrupy.extensions.json import JSONSnapshotExtension


class Entity(BaseModel):
    type: str
    text: str
    confidence: float = Field(ge=0, le=1)


class Result(BaseModel):
    entities: list[Entity]


def normalise(raw: dict) -> dict:
    parsed = Result.model_validate(raw)
    return {
        "entities": sorted(
            (
                {"type": e.type, "text": e.text, "confidence": round(e.confidence, 1)}
                for e in parsed.entities
            ),
            key=lambda e: (e["text"], e["type"]),
        )
    }


def test_press_release(snapshot):
    raw = extract_entities(fixture("press-release-2024-11.txt"))
    Result.model_validate(raw)
    assert normalise(raw) == snapshot.use_extension(JSONSnapshotExtension)

Same two assertions, same normalisation, and the JSON extension writes one readable file per case. Pydantic’s validation error is more informative than a snapshot mismatch would be, which is the point of keeping the schema check first and separate.

Where this still breaks

Structured output narrows the noise; it does not eliminate it. Free text inside a field is still free text, so an object with a summary string has the original problem confined to one key — either drop that key from the snapshot or assert a property of it rather than its value. A model that returns a variable number of entities produces array-length diffs that are real content changes and need reviewing, not normalising away. And a schema that permits an optional field gives you two shapes to record, so the snapshot is recording which branch the model took, which is usually worth knowing and occasionally the whole bug.

There is also a failure that looks like success. A model that cannot answer will often still return a schema-valid object — an empty array, a null field, a fallback enum member — and both assertions pass. Guard it explicitly rather than hoping: assert that the entity array is non-empty on cases where you know entities exist, and treat a sudden increase in empty results across the corpus as a regression signal in its own right. Schema conformance measures whether the response fits the box, and a box can be correctly shaped and empty.

The failure this setup does not catch at all is a correct-shaped wrong answer on a case nobody wrote. That needs a labelled dataset rather than a golden file — see testing structured output for the assertions that hold without one.