VCR Cassettes for LLM Tests in Python

A cassette is a YAML file holding one real request and one real response. Record it once, replay it forever, and your test suite stops costing money and stops being non-deterministic. The part that goes wrong is not the recording. It is the matching rule that decides whether an incoming request is the one on the tape.

Decide what you are asserting first

A cassette freezes one model output, which makes it tempting to assert on that output word for word. Do not. The moment you re-record, the assertion breaks for a reason that has nothing to do with your code, and you will “fix” it by pasting in the new sentence. That is a test that can only ever agree with itself.

What a cassette is genuinely good at is the code either side of the model. Assert that your request builder produced the right model, the right number of messages, the right tool schema. Assert that your parser turns this specific response into the right object — that it reads choices[0].message.tool_calls and not choices[0].message.content when the model chose a tool, that it handles a finish_reason of length differently from stop, that it does not explode on a null content field. The model output in the cassette is a fixture for your parser, not a claim about model quality.

Everything below assumes that split. If you find yourself wanting to assert on prose, what you actually want is an eval, which is a different tool with a different cadence.

The setup that works with the modern SDKs

VCR.py patches HTTP libraries, not SDKs. Its compatibility list covers http.client, requests, urllib3, aiohttp, httpx and httpcore, which is what matters here: the official OpenAI and Anthropic Python SDKs are built on httpx, so VCR.py sees their traffic without any injection or wrapper on your side. That is the main reason to reach for it over a requests-only mocking library. The VCR.py installation page carries the current list.

# tests/conftest.py
import vcr

llm_vcr = vcr.VCR(
    cassette_library_dir="tests/cassettes",
    record_mode="once",
    filter_headers=[("authorization", "REDACTED"), "openai-organization"],
    decode_compressed_response=True,
)

# tests/test_summariser.py
from myapp.summarise import summarise

@llm_vcr.use_cassette("summarise_short_doc.yaml")
def test_summariser_extracts_the_bullet_list():
    result = summarise("...three paragraphs of input...")
    assert result.bullets            # the parser found a list
    assert result.truncated is False # finish_reason was "stop", not "length"

decode_compressed_response=True is worth setting on the first day rather than the day you need it. Providers gzip responses, and a cassette holding a gzipped blob is unreadable in a diff, so you lose the single best property of the format — that a reviewer can see what changed.

match_on, and the gotcha

By default VCR.py decides two requests are the same one if the method, scheme, host, port, path and query agree. Notice what is not in that list: the request body. Every chat completion your application makes goes to the same POST /v1/chat/completions on the same host, so under the default rule they are all the same request, and a cassette with one interaction will answer all of them.

That is fine for a single-call test and wrong for anything that makes two different calls. The obvious repair is to add body to match_on, and this is where the trouble starts, because the body an SDK sends is not the body you wrote. It carries defaults the client filled in, and any part of your prompt that varies — a timestamp in a system prompt, a UUID, a retrieved document, a conversation id — varies in the body too. Adding headers is worse still: the OpenAI and Anthropic clients send per-request telemetry headers and an idempotency key, so no two requests ever have identical headers, and a header-matched cassette matches nothing.

What happens next depends entirely on the record mode, and this is the silent failure the page exists for. Under new_episodes, a request that fails to match is not an error — VCR.py goes to the network, gets a real answer, and appends it to the cassette. The suite passes. The file grows by one interaction on every run. You are billed on every run, and nothing tells you, because the only visible symptom is a fixture file with a suspiciously large diff.

The fix is to match on the part of the body that identifies the call and nothing else. Register a custom matcher and put it in match_on alongside the defaults:

import json

def match_on_model_and_last_message(r1, r2):
    b1 = json.loads(r1.body or b"{}")
    b2 = json.loads(r2.body or b"{}")
    return (
        b1.get("model") == b2.get("model")
        and b1.get("messages", [{}])[-1].get("content")
        == b2.get("messages", [{}])[-1].get("content")
    )

llm_vcr.register_matcher("llm_call", match_on_model_and_last_message)
llm_vcr.match_on = ["method", "scheme", "host", "port", "path", "llm_call"]

Now two different prompts get two different interactions and one repeated prompt reuses one, which is what people assume happens by default.

Record modes, and the one CI should use

• once — record if the cassette does not exist, replay if it does, and raise if a request arrives that the cassette cannot answer. The right default locally.
• none — replay only. Any unmatched request is an error rather than a network call. This is what CI should run, and it is the only setting that makes “no test in this suite talks to a provider” a fact rather than a hope.
• all — ignore what is on the tape and re-record everything. This is the refresh button, run deliberately.
• new_episodes — replay what matches, record what does not. Convenient while writing, and the mode that quietly bills you when your matcher is wrong. Do not leave it on.

Setting none in CI has a second benefit: a test that someone added without a cassette fails loudly in the pipeline instead of passing on a developer machine that happens to have a key in its environment. That is one of the named causes on the page about tests that pass locally and fail in CI, running in reverse.

Where a cassette stops helping

A cassette is a recording of one moment. It cannot tell you that the provider has since renamed a field or retired the model you recorded against — your suite stays green against an API shape that no longer exists, which is the whole subject of refreshing stale fixtures. It also stores a streamed response as a single body, so the chunk boundaries that your SSE parser actually has to handle are gone by the time you replay; see recording a streaming response as a fixture for what to do instead.

And a cassette records exactly what went over the wire, including your key. filter_headers only applies at record time, so a cassette taped before you added it still has the secret in it. If that has already happened, start from rotating the key, not from editing the file.