Snapshot Testing Python LLM Output With syrupy

syrupy gives pytest a snapshot fixture that reads as an ordinary assertion. The part worth learning for model output is not the fixture — it is the matcher and filter arguments that decide which parts of a response are allowed to move.

The fixture and where files land

syrupy adds a snapshot fixture and overloads equality against it, so the assertion is assert actual == snapshot. On first run the value is recorded; afterwards it is compared. Files are written under a __snapshots__ directory next to the test module, so tests/test_classifier.py gets tests/__snapshots__/test_classifier.ambr in the default Amber format, which holds every snapshot from that module in one readable file.

For model responses the JSON extension is usually a better default. It is a single-file extension: each test gets its own .json file, which means a regeneration touching forty tests produces forty separate diffs rather than one large one, and a reviewer can see at a glance how many cases moved.

# tests/conftest.py
import pytest
from syrupy.extensions.json import JSONSnapshotExtension


@pytest.fixture
def snapshot_json(snapshot):
    return snapshot.use_extension(JSONSnapshotExtension)

Normalise before you assert

The same rule as anywhere else in this cluster applies: do not put the raw completion in the snapshot. Reduce the response to the decisions you care about, and let the prose stay out of the file entirely.

# tests/test_classifier.py
from app.classifier import classify


def summarise(res) -> dict:
    msg = res.choices[0].message
    return {
        "model": res.model,
        "finish_reason": res.choices[0].finish_reason,
        "tool_calls": [t.function.name for t in (msg.tool_calls or [])],
        "label": res.parsed.label,
        "confidence_bucket": bucket(res.parsed.confidence),
        "pii_redacted": "@" not in (msg.content or ""),
    }


def test_double_charge_ticket(snapshot_json):
    res = classify("My card was charged twice this morning.")
    assert summarise(res) == snapshot_json

confidence_bucket is the interesting field. A raw float from a model changes on every call and is worthless in a golden file; the bucket it falls into is stable and is the thing a routing decision actually reads. Snapshot the quantity your code branches on, not the quantity the provider happened to return.

pii_redacted is the other field worth copying. It is a boolean derived from the output rather than a value read out of it, and derived booleans are the most valuable thing a golden file can hold: they turn a property you care about into something a diff can show. The same pattern extends easily — whether a citation marker appeared, whether the answer stayed inside a length bound, whether a currency symbol matched the locale in the request. Each is one line in summarise and each fails visibly.

Keep summarise in the test package rather than in application code. It is a description of what you are asserting on, and the moment it lives in app/ somebody will reuse it in production and then be unable to change it without breaking tests it was written to serve.

Matchers and exclude filters

When a volatile field must stay in the recorded value, syrupy takes two keyword arguments on the fixture call. matcher rewrites values during serialization; exclude drops keys entirely. path_type from syrupy.matchers maps a dot-delimited path to the types it may hold and writes the type name into the file instead of the value, and props and paths from syrupy.filters remove keys by name or by path.

from syrupy.filters import props
from syrupy.matchers import path_type


def test_double_charge_ticket(snapshot_json):
    res = classify("My card was charged twice this morning.")
    assert res.model_dump() == snapshot_json(
        matcher=path_type({"latency_ms": (float,), "usage.total_tokens": (int,)}),
        exclude=props("id", "created"),
    )

Prefer matcher to exclude where you can. An excluded key is invisible: if the provider stops returning usage altogether, an exclusion says nothing and a type matcher fails. Exclusion is for fields whose absence genuinely does not matter, such as a request-scoped id you never read.

Both arguments take paths into the serialized structure, which means they are coupled to the response shape. When a provider renames a field, a matcher whose path no longer resolves simply stops applying, and the volatile value it was suppressing reappears in the snapshot as a diff. That is the right failure — noisy and immediate — but it is worth recognising it for what it is rather than reaching for the update flag. If you find yourself writing more than a handful of paths, that is a signal to snapshot a reduction like the one above instead of the raw response object, because the reduction is a shape you own and the provider cannot rename it underneath you.

Approving an intentional change

The workflow that makes snapshot testing honest is the one where a person decides. syrupy has no separate approve command; the decision is expressed by running the update flag and then reviewing the resulting diff in version control, which is the same shape as approval testing.

1. Change the prompt or the code on a branch, and commit that change alone. No snapshot files in this commit.
2. Run pytest with no flags and read the failures. The count is information: a one-clause prompt edit that moves every snapshot in the suite usually means you changed the output format, not the behaviour.
3. Run pytest --snapshot-update. This rewrites snapshots to match the current assertions and deletes ones no longer used.
4. Commit the regenerated files as a second commit that touches nothing else, so the diff a reviewer opens is only golden files.
5. Review that diff field by field against a checklist, then run pytest again with no flags twice. A snapshot that fails on the second clean run is not deterministic and should never have been an exact snapshot.

Unused snapshots and CI

Because --snapshot-update deletes unused snapshots, running it against a filtered selection is dangerous: a run limited by -k or by a marker considers every unselected test’s snapshot unused. Regenerate against the full suite, or accept that you will be restoring files from git.

Two flags help in review. --snapshot-details puts the test name and location of each unused snapshot in the report, which turns “3 snapshots unused” into something actionable. --snapshot-warn-unused downgrades unused snapshots from a failure to a warning, which is right for a transitional branch and wrong as a permanent setting, because an accumulating pile of orphaned files is how a golden corpus becomes unreviewable.

In CI, run pytest with no snapshot flags at all. syrupy writes missing snapshots on a normal run, so a pipeline that executes a brand-new test for the first time will record whatever it saw and pass — the same hazard Jest addresses with its CI flag. The durable fix is not a flag but a rule enforced by version control: snapshot files may only enter the repository through a commit a person made, and a job that finds the working tree dirty after a test run fails the build. A one-line git diff --exit-code tests/__snapshots__ at the end of the test stage does that, and it catches the case where a test wrote a snapshot nobody intended regardless of which flag was passed.