When to Record a Cassette and When to Hand-Write a Fixture

Both approaches remove the network from your tests, and the argument for each is usually made in terms of fidelity versus convenience. That framing produces long debates and no rule. There is a better question, and it decides almost every case in one line.

The question that decides it

Could you have written this response from the documentation?

If no — if the exact bytes, the framing, the header set or the field ordering are things you would have got wrong by inventing them — record it. The value of a cassette is that it is evidence. If yes — if the response is a two-field object you fully understand and the interesting part is what your code does with it — hand-write it, because then the fixture doubles as documentation of the case, and a reader can see the whole scenario without opening a second file.

Underneath that question is a distinction about what is under test. Sometimes the provider’s behaviour is the subject: you are checking that you parse what it really sends. Sometimes your reaction is the subject: you are checking that a truncated JSON body takes the repair path, and where that body came from is irrelevant. Recording the second kind is an expensive way to obtain a value you could have typed, and it hides the case you are testing inside a large file nobody reads.

Record when the response is the subject

• Streaming. Server-sent event framing is full of details you will get wrong from memory: which event types appear in which order, where the chunk boundaries fall relative to token boundaries, whether a JSON value is split across two deltas, how the stream terminates, and what an error mid-stream looks like after a 200 has already been sent. Anthropic’s documentation notes explicitly that a streaming error arrives after the 200 and does not follow the normal error mechanism — that is precisely the kind of thing a hand-written fixture will not contain because you did not know to put it there.
• Error responses. Real ones carry retry-after, rate-limit headers and request ids, and the documented example body is usually cleaner than what arrives. See maintenance-window handling for a case where the header is the whole test.
• Anything you are surprised by. If a response made you say “it does what?”, record it now. That is a behaviour you cannot reconstruct later, and next quarter you will not believe your own note about it.
• Multi-step interactions. A tool-calling turn followed by a result submission followed by a final answer has correlation ids threading through it, and recording the sequence keeps them consistent for free.
• A bug report you cannot reproduce on demand. The recording is the only copy of the evidence, which makes it worth keeping even when it is awkward — subject to surviving redaction.

Hand-write when your reaction is the subject

• Edge cases you would struggle to provoke. An empty completion, a response with content and no tool call when your code expects one, a tool call whose arguments are valid JSON but violate your schema, a finish reason you have never seen. Waiting for a real provider to produce these is not a test strategy; typing them takes a minute each.
• Boundary values. Output stopping exactly at max_tokens in the middle of a JSON string, a response one token over a limit, a zero-length message. You need the exact value, and a recording gives you whatever the model happened to do.
• Combinatorial coverage. Twelve variations of one shape belong in a table-driven test with twelve small literals, not in twelve cassettes. The table is readable; the cassettes are twelve files.
• Anything a reviewer needs to understand at a glance. A fixture inline in the test is part of the argument the test is making. A fixture in a separate 400-line YAML file is a footnote nobody follows.

What each one costs you later

Both have a maintenance profile, and knowing them is most of the decision when a case could go either way.

A cassette rots. It was true on the day it was recorded and nothing tells you when it stops being true, which is why it needs an active detector — a scheduled re-record and shape diff, as in detecting cassette drift. It also carries secrets by default, so header filtering has to be configured before the first recording rather than after the first leak, and it is verbose enough that a review of a large recorded diff is not a real review. And matching is subtle: VCR.py matches on the URI and method by default, so a cassette will happily answer a request whose body you have changed since recording.

A hand-written fixture drifts in a different way. It never rots, because it was never true — it encodes your belief about the API, and if that belief was wrong the test proves you are consistent with your own misunderstanding. The mitigation is to derive hand-written fixtures from a recorded one: record a real response once, then write your small literals to match its actual shape, and keep one contract test that checks the real shape still matches. That gives you readable fixtures with a single point of contact with reality.

Using both in one suite

The arrangement that works in practice is a thin layer of recordings and a thick layer of literals. A handful of cassettes covering the transport — one streaming response per provider, one of each error class, one full tool-calling round trip — and everything else hand-written against the shapes those recordings establish. That keeps the number of files that can rot small enough to actually monitor, and keeps the tests that encode your logic legible.

Two rules make the split hold. Recordings live in one directory with the drift job pointed at it, so nobody has to remember which files are recorded. And a hand-written fixture that starts growing — past thirty lines, or acquiring fields nobody asserts on — is a sign it wants to be a recording, or more often that the test wants splitting. Neither format is a place to accumulate detail for its own sake.

There is a third option that is not a fixture at all and is sometimes the right one: for tests about your own orchestration — retry policy, budgets, routing, concurrency — a stub that returns a hard-coded value with no attempt at realism is clearer than either, because the point is the control flow. That case is worth reading separately in testing without the model, and for replaying real traffic at volume rather than case by case, traffic replay is the related technique.