Mocking an LLM API With responses in Python

responses is the lighter option when you would rather write the model response by hand than record one. Before you write a line of it, check one thing, because there is a very common case where it cannot work at all and the failure is confusing.

Check which HTTP library you are mocking

The responses library is, in its own description, a utility for mocking out the requests library. It patches requests’ transport adapter. It does not patch httpx, aiohttp or urllib3 used directly.

The official openai and anthropic Python SDKs are built on httpx. So if your code calls client.chat.completions.create(...), responses will register your mock, never intercept anything, and let the call go to the real network — or fail with an authentication error in CI, which at least is loud. The registered mock then reports that it was never fired, if you asked it to, and that message is the clue.

• Your code calls an OpenAI-compatible endpoint with requests — a thin wrapper around a vLLM, Ollama, llama.cpp or gateway endpoint, which is extremely common — then responses is exactly right and the rest of this page applies.
• Your code uses the official SDK — use respx, which does the same job for httpx, or VCR.py, which patches httpx among others.

The equivalent in respx is close enough that translating is mechanical: a route is declared with respx.post(url, json__eq={...}) and given a response with .mock(return_value=httpx.Response(200, json=...)), and streaming is expressed by passing an iterable of byte chunks. The respx API reference has the current lookup syntax.

The stub

import responses
from myapp.client import summarise

URL = "http://localhost:8000/v1/chat/completions"

COMPLETION = {
    "id": "chatcmpl-abc",
    "object": "chat.completion",
    "model": "Qwen2.5-7B-Instruct",
    "choices": [
        {
            "index": 0,
            "finish_reason": "stop",
            "message": {"role": "assistant", "content": "Three bullet points."},
        }
    ],
    "usage": {"prompt_tokens": 214, "completion_tokens": 9, "total_tokens": 223},
}

@responses.activate
def test_summariser_parses_the_first_choice():
    responses.post(URL, json=COMPLETION, status=200)

    assert summarise("some input") == "Three bullet points."

    sent = responses.calls[0].request
    assert "Bearer " in sent.headers["Authorization"]

Hand-writing the fixture is the point of choosing this over a recording. It lets you write the response you need to test rather than the one the model happened to give you — a finish_reason of length, a content of null next to a tool call, a usage block missing a field, an empty choices array. Those are the shapes your parser gets wrong, and you will wait a long time for a live model to produce them on demand. That is the argument made at length in unit-testing an LLM output parser.

Two details in that fixture are load-bearing. finish_reason is present because a completion that stopped at the token limit is a different outcome from one that finished, and a summariser that returns a sentence cut in half without saying so is a worse failure than one that raises. And usage is present because if your code records token counts for cost attribution, a fixture without it means that code path is never run in a test and a missing key will surface as a production exception rather than a red build.

Keep the fixtures as module-level constants and derive variants from them with a shallow copy rather than pasting a second full response. Otherwise the truncation test and the happy-path test drift apart, and when the wire format changes you have to find every copy.

Matching the request

By default a registered response matches on method and URL, so a test making two different completions gets the same answer twice. Add matchers from responses.matchers to distinguish them:

from responses import matchers

@responses.activate
def test_classifier_and_summariser_use_different_prompts():
    responses.post(
        URL,
        json=CLASSIFICATION,
        match=[matchers.json_params_matcher({"model": "small"}, strict_match=False)],
    )
    responses.post(
        URL,
        json=COMPLETION,
        match=[matchers.json_params_matcher({"model": "large"}, strict_match=False)],
    )

    assert classify("some input") == "billing"
    assert summarise("some input") == "Three bullet points."

The strictness flag is the field to look up rather than assume. By default the JSON matcher compares the whole decoded body, and a chat request carries fields you did not name, so an unrelaxed matcher matches nothing. Check the signature for the version you have pinned; the responses repository documents the matchers, and the flag names have moved between releases.

If matching a body precisely turns into a fight, do what the gock page recommends: match loosely, then assert on responses.calls[0].request.body in plain Python. A failed assert gives you a diff; a failed match gives you a connection error.

Matching on headers has a use here beyond assertion. If your application talks to more than one endpoint — a small model for classification and a large one for generation, or a local server alongside a hosted provider — then distinguishing the stubs by URL or header rather than by body keeps each one readable, and makes a misrouted request fail with a clear message instead of quietly receiving the other model’s answer.

Counts, order and retries

The assertions that catch real bugs here are about how many times you called, not what came back.

• responses.assert_call_count(URL, 1) proves a cache or a deduplication actually prevented a second billed call. An “at least once” assertion cannot.
• RequestsMock(assert_all_requests_are_fired=True) as a context manager turns “this mock was never used” into a failure. That is the assertion that would have caught the httpx problem at the top of this page on the first run.
• For retry tests you need a sequence — a 429 then a 200. The library supports ordered registries for exactly this; register the failure first and the success second, and assert the call count is two. Look up the registry class name for your version rather than copying one, as the registries API is newer than most examples on the web.

Pair the count assertion with an assertion on elapsed behaviour rather than elapsed time: inject the sleep function so the test can record that a two-second backoff was requested instead of spending two seconds.

When to record instead

Hand-written fixtures drift from reality in a way recordings do not: nothing ever forces you to check that the shape you typed still matches what the provider sends. A recording at least gives you an artefact with a date on it and a diff when you re-record.

A reasonable split is to hand-write the awkward cases — the truncations, the refusals, the malformed tool arguments — because those are hard to capture live, and to record the happy path so that a real change in the wire format shows up when you refresh it. Keeping both in the same suite is fine as long as it is obvious which is which: a directory name is enough, and it stops somebody re-recording over a fixture that was written by hand precisely because no live call produces it. What you must not do is assume either one stays true; stale fixtures are how a green suite ends up testing an API that no longer exists.