When a Cassette Goes Stale: Refreshing Recorded LLM Fixtures

A recorded fixture is a claim about how a provider behaved on the day you recorded it. Nothing in your test suite ever re-checks that claim. That is not a flaw in the tooling; it is the entire reason recording makes tests fast and free, and it means the failure mode is permanent and silent rather than occasional.

Why a green suite proves nothing here

Consider what a replayed test actually verifies. Your code builds a request; the cassette layer matches it against a saved request and hands back a saved response; your code parses that response. Both ends of the loop are yours. The provider is not in the loop at all.

So the suite verifies that your parser can read a response the provider sent last November. If the provider changed the response in March, the suite is still green and your production code is broken. There is no test you can add inside the replayed suite that catches this, because the information is not present. The only way to learn it is to talk to the provider, which is exactly the thing recording exists to avoid.

This is worth stating plainly because the usual instinct — write a stricter assertion — makes it worse. A stricter assertion against a stale fixture is a more confident wrong answer. The fix is structural: keep the fast replayed suite, and add a small, separate, scheduled thing that does talk to the provider.

What actually changes underneath you

The changes that break a recorded suite are rarely dramatic. They are mostly additive or peripheral, which is why they slip past.

• A model id is retired. Your fixture answers happily to a model name that now returns a 404 or a deprecation error in production. Nothing in the replay path knows the name is dead.
• A field appears, moves or is renamed. Reasoning tokens, cache-read token counts, refusal fields and annotation blocks have all been added to completion responses at various points. Your parser is untested against them because your fixture predates them.
• A request parameter starts being rejected. A parameter that was ignored becomes an error, or becomes mutually exclusive with another one. The replayed test never sends the request anywhere that could reject it.
• An enum gains a value. A new finish_reason or a new error type that your switch statement falls through on. This one usually surfaces as a production null.
• The model behind a stable name changed. Not a wire format change at all, and invisible to any structural test — see silent model updates.

Note that the last one is a different problem wearing the same coat. A stale fixture means your recorded bytes no longer match reality. A stale expectation means the bytes are fine and your assumption about quality is not. Recording tools address the first and are silent about the second; evals address the second. Conflating them produces a suite that is strict about the wrong half.

Make the age of a fixture visible

The first mechanism is cheap: make staleness a thing you can see and fail on. File modification time will not do it — a fresh git clone stamps every file with the checkout date, so every CI run sees fixtures that look one minute old.

Use the date inside the artefact instead. VCR.py writes a recorded_at timestamp into each interaction in the cassette YAML, so a small test can walk the fixture directory, read the oldest one and fail — or warn — past a threshold you choose. For formats that do not carry a date, add one yourself at record time and commit it alongside: the point is that the number travels with the file, not with the filesystem.

Choose the threshold from the pace of the thing you recorded, not from a round number. A fixture of a stable, versioned, dated model endpoint can sit for a long time. A fixture of a floating alias, a preview feature or a beta header is a shorter-lived object and should be flagged sooner.

A small live tier, on a schedule

The second mechanism is a handful of tests — not the suite, a handful — that run against the real provider on a schedule rather than on every commit, and that assert only on shape. Does the endpoint still accept this request? Does the response still contain choices[0].message? Is usage still present with the field names we read? Is this model id still served?

Keep them cheap and keep them structural. One completion with a two-token output per provider per day costs almost nothing and answers the only question a replayed suite cannot. Do not assert on content: that turns a wire-format check into a flaky quality check and it will be muted within a month, which is the usual reason this tier does not exist in projects that once had it.

Run it on a schedule rather than in the pull-request pipeline. In a PR it is a network dependency that fails for reasons unrelated to the change and trains people to re-run red builds. On a nightly or weekly trigger it is a signal with a clear owner. The general shape of this argument, and where the boundary between mocked and live testing sits, is in testing without the model.

Refreshing without rewriting the suite

1. Re-record deliberately, into the same files, with the tool’s refresh mode — VCR.py’s all record mode, nock’s update fixture mode. Delete-and-re-record works too but loses the diff, which is the valuable part.
2. Read the diff. This is the step people skip and it is where the information is. A new field, a changed field order, a different error envelope, a token count that moved: each of those is either a change your parser must handle or a change you can ignore, and you have to decide which.
3. Fix the code before touching the assertion. If a test now fails against a fresh recording, the default assumption is that your code is wrong, not that the test is. Editing an assertion to match a new recording is how a suite becomes decorative.
4. Re-run the redaction check. A refresh is a new recording and it captured your key again; redacting keys from cassettes has the pre-commit check that makes this automatic.
5. Commit the refreshed fixtures in their own change, separate from feature work, so the diff is reviewable rather than buried.