Isolating Flakiness Caused by the Provider From Flakiness Caused by Your Code

“Is it us or is it them?” is answerable in about ten minutes, and the answer decides everything about what you do next. The procedure is two replays of the same recorded request, one holding the response fixed and one holding the request fixed.

The boundary to cut at

There is exactly one clean seam in a model-backed application: the HTTP request that leaves your process and the response that comes back. Everything upstream of the request bytes is your code — prompt templating, retrieval, serialisation, parameter selection. Everything between the request bytes and the response bytes is the provider. Everything downstream of the response bytes is your code again — parsing, validation, the assertion.

That seam is useful precisely because it can be frozen in either direction. Freeze the response and vary nothing else: any remaining variance is yours. Freeze the request and send it repeatedly: any variance is theirs. Neither experiment needs the other, and running both is what makes the result a diagnosis instead of a guess.

Cut anywhere else and the result is ambiguous. Mocking your own client class rather than the transport means you are also mocking your serialisation, so a bug in how you build the request becomes invisible — and building the request is where a surprising share of these bugs live.

Capturing the exact request

You need the request bytes as they went on the wire, not as you believe you constructed them. Record at the transport layer. In Python that is VCR.py or the responses library; in TypeScript, MSW’s Node server. All three intercept below your SDK, which is the property that matters.

# capture once, with the key stripped, matching on the body so a prompt
# change cannot silently replay the wrong recording
import vcr

recorder = vcr.VCR(
    cassette_library_dir="tests/cassettes",
    record_mode="all",
    match_on=["method", "host", "path", "body"],
    filter_headers=["authorization", "x-api-key"],
    filter_post_data_parameters=["api_key"],
)

with recorder.use_cassette("router_tool_call.yaml"):
    run_the_failing_case()

The match_on list is the part people get wrong. Its default does not include the request body, so two requests to the same URL with entirely different prompts match the same recording — which means replay A can appear stable for the excellent reason that it is replaying the wrong interaction. Adding body makes a changed prompt a loud cassette miss rather than a quiet wrong answer, and a quiet wrong answer here corrupts the whole diagnosis that follows.

Commit the cassette. It is the artefact the whole diagnosis rests on, and a recording that only exists on one laptop cannot settle an argument between two people. Check that filter_headers did its job before committing — open the YAML and look.

The two replays

Replay A — fixed response, your code runs. Set record_mode="none" so nothing can reach the network, and run the test twenty times. The model’s output is now a constant. If the test still fails intermittently, the model is not involved: the variance is in your parsing, your assertion, a dictionary ordering, a timestamp, a random seed, or shared state between tests.

# TypeScript equivalent with MSW v2
import { http, HttpResponse } from "msw";
import { setupServer } from "msw/node";

const server = setupServer(
  http.post("https://api.example.com/v1/chat/completions", () =>
    HttpResponse.json(recordedResponse),
  ),
);

beforeAll(() => server.listen({ onUnhandledRequest: "error" }));
afterAll(() => server.close());

onUnhandledRequest: "error" is the equivalent of record_mode="none": it turns any request you did not stub into a loud failure instead of a real network call. Without it the experiment is not controlled, because one un-stubbed endpoint reintroduces exactly the variance you are trying to eliminate.

Replay B — fixed request, the provider runs. Take the recorded request body and send it directly, twenty times, outside your test framework entirely. Nothing of yours executes except an HTTP client and a comparison.

#!/usr/bin/env bash
# replay the captured body 20 times and summarise what varies
for i in $(seq 1 20); do
  curl -s -o "/tmp/resp-$i.json" -w "%{http_code} " \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -H "content-type: application/json" \
    --data @recorded-request.json \
    https://api.openai.com/v1/chat/completions
done
echo
jq -r '.model, .system_fingerprint, (.choices[0].finish_reason)' /tmp/resp-*.json \
  | sort | uniq -c | sort -rn

Do not compare the completion text — it is expected to vary. Compare the fields that are supposed to be stable: the status code, the served model, the system_fingerprint where the provider returns one, and finish_reason. A varying model id means an alias resolving to more than one version. A varying finish_reason — some responses coming back as length rather than stop — means your max_tokens is marginal for this prompt, which is a your-code problem masquerading as provider noise.

The truth table

Replay A          Replay B          diagnosis
(fixed response)  (fixed request)
----------------  ----------------  ---------------------------------------
stable            stable            not reproducible at the HTTP boundary.
                                    Look at the harness — see below.
flaky             stable            your code. Parsing, ordering, shared
                                    state, an unseeded value, a clock.
stable            varies            the provider or the model. Pin the
                                    version, widen the assertion, or accept
                                    it and use N-of-K.
flaky             varies            both, and they are independent. Fix the
                                    deterministic side first: it is cheaper
                                    and it may be all of it.

The third row is the one people expect and the second row is the one that actually turns up most often, which is the reason the procedure is worth running rather than reasoning about. Once you are in row three, the options are the rest of this cluster: assert on a property instead of the text, require K of N passes, or quarantine with an owner and a date.

The third category: your harness

The first row of the table is the interesting failure, because it means the flakiness lives in neither the model nor the code under test but in the machinery around them. Four causes account for most of it:

• Test-order dependence. One test leaves state behind — a cached client, a patched module, a populated fixture directory — and the next test’s outcome depends on ordering. Confirm it by fixing the order: pytest with -p no:randomly, or Vitest with sequence.shuffle disabled. If a fixed order is stable and a random one is not, this is your bug.
• Concurrency against a shared limit. Parallel workers sharing one API key hit a per-minute limit that a single worker never reaches. This looks like model flakiness and is arithmetic; the locally-green, CI-red page covers the diagnosis.
• Timeouts that are marginal. A per-test timeout close to the model’s p95 latency converts ordinary latency variance into a flaky test. The tell is a failure with no assertion in it.
• The clock. Anything interpolating the current date into a prompt makes yesterday’s recording no longer match today’s request. Freeze it in a fixture.