Mocking and Recording LLM Responses in Tests

Cassette libraries record an HTTP interaction once and replay it forever. The pattern transfers to model calls almost unchanged, and the one thing that does change — that you cannot re-run the call to get the same bytes — is what makes it more valuable here than it ever was for a REST API.

Why record at all

A hand-written fake is better than a cassette for testing logic: it is explicit, it is readable, and it does not drift. A cassette is better for exactly one thing, which is fidelity. Real responses contain the details you would never think to invent — the exact usage field layout, the ordering of streamed chunks, the way a tool call is split across deltas, the unusual finish reason, the whitespace before the JSON.

So use both, for different jobs. Scripted fakes for behaviour: retries, deadlines, breakers, business rules. Cassettes for the boundary: adapter parsing, streaming assembly, usage accounting. If a test is asserting something about your logic, a cassette makes it harder to read for no benefit.

There is a second reason specific to this dependency. With a deterministic API you can always regenerate a fixture by calling again. Here you cannot: the same request produces different bytes tomorrow, and the model behind the name may be replaced entirely. The recording is therefore not a convenience — it is the only reproducible artefact you will ever have of that interaction, which means it belongs in version control and deserves review.

The matcher is the design

A cassette is a map from request to response, and the whole quality of the setup is in how a request is turned into a key. Match on too much and every trivial edit misses the cassette; match on too little and a test gets someone else’s recording and passes for the wrong reason.

function cassetteKey(req: ProviderRequest) {
  return sha256(canonicalJson({
    endpoint: req.path,
    model: req.body.model,
    messages: req.body.messages,      // the prompt IS the identity of the call
    tools: req.body.tools ?? null,
    // Deliberately excluded, because none of these change the answer we
    // are recording, and including them makes cassettes miss for no reason:
    //   authorization / api-key headers, user-agent, request id,
    //   idempotency key, trace headers, timestamps injected by our own code.
  }));
}

Sensible defaults: match on method, path, model and the full message array; ignore all headers except content type; and canonicalise JSON (sorted keys, stable number formatting) so that a serialisation change does not invalidate every cassette in the repository.

One trap deserves its own sentence. If any part of your prompt contains a timestamp, a UUID, a random sample or a locale-dependent format, the key changes on every run and no cassette will ever match. The fix is not to relax the matcher but to make the non-determinism injectable — the clock and the random source are parameters, and in tests they are fixed. This is worth doing regardless; a prompt that varies for reasons unrelated to the input also defeats every caching layer you have.

Recording a stream

A streamed response is not one body, and flattening it to the concatenated text discards precisely the thing worth testing. Record the chunk sequence.

{
  "key": "5f2c...",
  "recorded_at": "2026-08-03T09:14:22Z",
  "status": 200,
  "stream": [
    { "at_ms": 412, "data": "{\"choices\":[{\"delta\":{\"role\":\"assistant\"}}]}" },
    { "at_ms": 431, "data": "{\"choices\":[{\"delta\":{\"content\":\"The\"}}]}" },
    { "at_ms": 448, "data": "{\"choices\":[{\"delta\":{\"content\":\" answer\"}}]}" },
    { "at_ms": 902, "data": "[DONE]" }
  ]
}

Keeping the relative timings gives you two things a flat body cannot. You can replay with the delays intact, which is the only way to test a time-to-first-token timeout, an inter-chunk idle detector or a UI that behaves differently when a stream stalls. And you can replay without them for speed, which is what you want in the ordinary suite. Make it a flag on the replayer.

Record the awkward cases deliberately, because they are the ones your assembly code gets wrong: a chunk that splits a multi-byte character, a tool call whose arguments arrive across six deltas, a stream that ends without a terminal event, and a stream that emits an error event after content. Those five cassettes will find more bugs than fifty happy-path ones.

Redaction, before it touches the disk

A cassette is a recording of a real request that carried a real credential, and it is about to be committed to a repository. Redact on the write path, never as a later cleanup, because the cleanup is what gets forgotten.

• Authorisation headers and API keys — replace with a fixed placeholder. Do not merely omit them, so that a diff makes the substitution visible.
• Anything in the prompt that came from real data. Recording against production data puts customer content into your repository permanently. Record against a fixture corpus instead; if you must record against real data, run a redaction pass and review it by hand.
• Account identifiers and organisation ids in response headers, which several providers include.

Back this with a repository-level secret scanner in a pre-commit hook. Redaction that depends on a person remembering is redaction that will fail once, and once is enough.

The re-record workflow

Cassettes rot. The provider changes a field, adds a value to an enum, alters how a tool call is chunked — and your tests keep passing against a recording of last year’s API while production breaks. This is the failure mode of the whole pattern and it needs a deliberate answer.

Two habits keep the rot visible. Stamp every cassette with the date and the model id it was recorded against, and fail — or at least warn loudly — when a cassette is older than some threshold you have chosen, so staleness is surfaced by the suite rather than by an incident. And run the re-record job on a schedule against a separate, budgeted key, so the cassette diff arrives as a pull request that a human reads. That diff is the earliest warning you will get that a provider changed something under you.