Skip to content
Multigrid

Snapshot and property testing for model output

Golden files, fuzzy matching, invariants and metamorphic relations — how to assert on text that is never byte-identical twice.

A test asserts that something is equal to something else. That instruction survives contact with a language model for about one afternoon: the same prompt, the same model and the same parameters return prose that is close in meaning and different in bytes, and a suite built on string equality starts failing for reasons that have nothing to do with your code. The usual reactions are both wrong. One is to delete the tests. The other is to keep them and re-approve every failure without reading it, which is the same thing with more ceremony.

The useful move is to change what you assert on. A schema holds when the wording does not. So does a tool name, a call count, a redaction, an ordering, a bound on length, and a relation between two runs that must hold whatever either one says. These pages work through that shift concretely — the snapshot machinery when a golden file is still the right tool, the mechanisms that make exact matching fail, and the properties and metamorphic relations that are testable precisely because the output is not.

Snapshot Testing LLM Output With Jest

How to write a Jest snapshot serializer that reduces a model response to the fields you meant to assert on, so the diff shows meaning rather than rewording.

9 min read

Snapshot Testing Python LLM Output With syrupy

Using syrupy's snapshot fixture, matchers and exclude filters on model responses, and the review workflow for approving a change on purpose.

9 min read

Approval Testing for Prompt Output

Approval testing names what golden-file suites for model output actually are: a record that a human read the new answer and signed it off.

8 min read

Why Exact-Match Snapshot Tests Fail on LLM Output

The five separate mechanisms that make the same request return different bytes, and why temperature zero does not remove any of them completely.

9 min read

Fuzzy Snapshot Matching for Non-Deterministic Text

Structural diffing, embedding similarity thresholds and key-fact extraction — what each technique can detect, and the specific failure each one is blind to.

9 min read

Updating Golden Files After an Intentional Prompt Change

A regeneration procedure that separates the change you meant from the regression you did not, instead of absorbing both in one commit.

9 min read

Reviewing a Snapshot Diff Before Approving It

A fixed order of checks for the two-minute decision on a golden-file diff, ordered so the changes that matter are seen before attention runs out.

8 min read

Snapshot Testing Structured JSON Output From a Model

Validate the schema on every run and snapshot the normalised values, so a diff shows a changed field rather than a reordered object.

9 min read

Storing Golden Files in Git Without Bloating the Repo

Why text goldens belong in Git and binaries do not, what Git LFS costs you in review, and when a manifest of content hashes is the better answer.

9 min read

Semantic Diff for Comparing Two Model Outputs in Tests

Four ways to score how far two generations differ in meaning, ordered cheapest first, and how to turn the score into an assertion you can defend.

10 min read

Property-Based Testing for LLM Output With Hypothesis

Generate the inputs with Hypothesis, keep the model as the system under test, and assert an invariant that holds across every generated case.

10 min read

Property-Based Testing LLM Output in JavaScript With fast-check

Building an async property around a model call in fast-check, and using the seed and path it prints to replay the exact failing case.

10 min read

Writing Invariants for LLM Output You Can Actually Test

Five families of property that hold for non-deterministic text, the ones that only look like properties, and how to derive yours from a spec you already have.

10 min read

Property-Based Testing a JSON Schema an LLM Must Satisfy

Two test directions around one schema: generate instances from it to harden your consumer, and generate prompt inputs to find where the model cannot satisfy it.

10 min read

Metamorphic Testing for LLM Prompts

How to test for correctness when there is no correct answer to compare against, by asserting a relation between the outputs of two related inputs.

10 min read

Testing That Paraphrased Inputs Get Consistent Answers

Implementing the paraphrase invariance relation end to end, including deciding what consistent means and separating paraphrase variance from sampling variance.

10 min read

Generating Test Inputs With a Grammar Instead of Examples

Writing a small context-free grammar for your inputs so a generator produces hundreds of realistic prompt variants, and keeping the derivation so failures are classifiable.

10 min read

Shrinking a Failing Property Test Case to Its Minimal Prompt

Why the minimal failing prompt is the deliverable rather than a nicety, how the two main shrinkers get there, and what non-determinism does to the process.

10 min read

Other topics