Recording Fixtures Once and Never Paying for the Same Test Twice

Record a real response once, replay it forever, pay nothing again. That is the pitch and the arithmetic genuinely is that good — but only if you also price the re-recording, because a cassette that is never refreshed is a test asserting against a world that has moved on.

The amortisation arithmetic

Take the same suite used in the CI cost estimate: 120 cases at a derived $0.01065 each, under the placeholder rates of $3.00 per million input tokens and $15.00 per million output tokens stated there. Recording the whole suite once costs $1.28. Running it live at 14 triggers a day over 22 active days is 308 runs, which the same figures put at $393.62 a month.

Live, every run
  308 runs x 120 cases x $0.01065        = $393.62 / month

Recorded once, replayed
  1 recording x 120 cases x $0.01065     =   $1.28 one-off
  308 replays                            =   $0.00

Recorded, re-recorded weekly
  1.28 x 4.3 weeks                       =   $5.50 / month

  [prices are the placeholder rates above — substitute your own]

That is the honest version: roughly $5.50 a month rather than $394, and the difference is not the whole $394 because the re-recording is the part that keeps the saving legitimate. A team that quotes the $1.28 figure and never re-records has not saved $392, it has stopped testing against the provider.

The other reason the arithmetic understates the win: replayed tests do not just cost nothing, they take milliseconds instead of seconds and they cannot be rate limited, cannot be flaky and cannot fail because a provider had an incident. On a suite of any size the wall-clock difference changes how often people run it locally, which is worth more than the money.

What a recording can and cannot prove

A cassette turns your model call into a fixed input. Everything downstream of that input is fully testable, deterministically and free.

• Provable from a recording: that your parser handles this response shape; that your schema validator accepts it; that business logic produces the right result from it; that a truncated, refused or malformed response is handled — record those cases deliberately, or hand-write them; that retry and error paths behave, by recording a 429 and a 500; that your token accounting adds up.
• Not provable from a recording: that the model still produces this response. By construction. The recording is a statement about one past moment, and no amount of replaying it tells you anything about the present.

This is the whole reason the pattern needs a second tier next to it. The recorded tier proves your code is correct with respect to a response; a small scheduled live tier proves the response is still what the provider sends. Neither is optional and neither is expensive — see splitting tests into a mocked tier and a live tier.

The maintenance cost nobody prices

A stale cassette fails in the worst available direction: it passes. The provider changed a field name, added a wrapper object, started returning a refusal where it used to answer, or deprecated the model entirely — and your suite is green, because it has not spoken to them in four months.

Three practices keep it honest, and all three are cheap. Re-record on a schedule rather than on demand, with a job that re-records into a branch and opens a pull request, so the diff of what the provider changed is reviewable rather than invisible. Store the recording date and the model string inside the cassette, and fail the test if the recording is older than a threshold you chose — a test that fails because its fixture is six months old is annoying exactly once and correct every time. And treat a re-recording diff as a review artefact: a changed cassette is the provider’s changelog for the part of the API you actually use.

The tooling is mature on both sides. In Python, VCR.py records HTTP interactions to YAML cassettes with a record mode that governs whether new requests may be made; pytest-recording exposes that through markers and a --record-mode flag. In JavaScript, Nock has a recorder, and MSW handles the playback side as a request-level mock. Check the current interface in each project’s own documentation before wiring it — these are the libraries whose surfaces have changed most.

Matching is the other thing to configure deliberately. A recorder decides whether an incoming request corresponds to a stored one using a matcher over the method, the URI and optionally the body, and the default rarely includes the body — which means every call to the same chat completions endpoint matches the first recording regardless of what you asked. For model calls the body is the request, so the matcher has to include it or the suite will replay one answer to every question and pass.

Redaction is not optional

A cassette is a verbatim recording of an HTTP exchange, which means by default it contains your Authorization header, and it is about to be committed to a repository. This has leaked real keys.

Configure header filtering before you record anything, not after — VCR.py takes a list of headers to filter and can also filter query parameters and post-data fields. Filter the request side and then check the response side too, since some providers echo organisation or project identifiers back in headers. Add a repository check that greps the cassette directory for anything shaped like a key on every commit; it is ten lines and it is the difference between a near miss and an incident.

The same applies to the payloads. Recorded prompts often contain real customer data pulled from a fixture database. Redact at record time, because a cassette committed once is in the history forever, and rewriting history across a team is a much larger job than filtering a field.

A cassette that is not committed

The single most common failure with this pattern has nothing to do with cost. Someone adds the cassette directory to .gitignore — often because the first version contained a key — and the suite passes locally, where the recordings exist, and fails in CI, where they do not. Worse, if the record mode allows new recordings, CI silently makes live calls instead of failing, and you are paying for a suite you believe is free.

Set the record mode to refuse new recordings in CI so an unmatched request is an error rather than a network call, and add one test that asserts the cassette directory is non-empty. Both are one line, and together they convert the failure from a silent invoice into a red build with an obvious cause.