Recording Real Provider Error Responses as Fixtures

Error-handling code is written against an imagined payload and then tested against the same imagined payload, so the test passes and production does not. The fix is boring and takes an afternoon: provoke each error once against the real API and keep what comes back.

Why a guessed error fixture is wrong

A hand-written error fixture is usually wrong in at least one of four ways, and each of them produces a different production bug.

• Nesting. Anthropic’s errors documentation shows a top-level envelope with type set to error, an error object carrying its own type and message, and a sibling request_id. Code that reads a top-level message, or that reads error.code because that is what another provider uses, gets undefined and logs an unhelpful line at exactly the moment you need a helpful one.
• The type vocabulary. The same documentation maps statuses to type strings: invalid_request_error at 400, authentication_error at 401, billing_error at 402, permission_error at 403, not_found_error at 404, conflict_error at 409, request_too_large at 413, rate_limit_error at 429, api_error at 500, timeout_error at 504 and overloaded_error at 529. A branch written against a guessed spelling never fires. See Anthropic’s errors reference for the current list, which the same page notes may expand.
• Headers. This is the big one. Retry logic reads retry-after, and diagnostics read the request id header. A fixture that is only a JSON body cannot exercise either, so a backoff implementation that honours retry-after is completely untested and a bug in it is invisible until a real rate limit arrives.
• Errors that arrive after a 200. On a streaming response the connection succeeds and the failure comes as an event in the stream. Anthropic’s errors page says exactly this: when receiving a streaming response over server-sent events an error can occur after a 200 is returned, and standard error handling does not apply. If every error fixture you own is a non-2xx status, your streaming error path has no test at all.

What to capture

Store four things per fixture, in a file named for the condition rather than for the status code: the status, the full response headers, the raw body bytes, and a small provenance block recording the provider, the endpoint, the date and the SDK version in use. Provenance is not bureaucracy — in eighteen months somebody will ask whether a fixture still reflects reality, and the recording date is the only thing that answers it.

Strip credentials at capture time, not later. Remove any authorization or API key header, any cookie, and any identifier belonging to a real customer that happened to be echoed in the request. This is the same discipline cassette-based tools apply through header filters, and the reason is the same: a fixture is a file in version control forever, and a credential in one has to be treated as leaked even if it was rotated.

Provoking each error on purpose

1. 401. Send a syntactically plausible but invalid credential. This is the easiest one and it costs nothing.
2. 400. Send a model id that does not exist, then send a request with a parameter the model rejects. Capture both: the messages differ and your code may want to distinguish them.
3. 404. Request a resource id that has never existed — a batch or file id with the right prefix and a random suffix.
4. 413. Send a body over the documented request size limit. Anthropic publishes 32 MB for the Messages and token counting endpoints, and notes that on the direct API this error is returned by Cloudflare before the request reaches the API servers — which is precisely why the response may not look like the others, and precisely why recording it beats guessing.
5. 429. The honest way is a key on a low tier driven past its limit with a short burst. If you cannot get one, do not fabricate the payload: use the documented type string, mark the fixture as hand-written in its provenance block, and record the real one the first time production sees it.
6. 500, 504 and 529. You cannot provoke these on demand. Capture them opportunistically: add a hook to your client that writes any unrecognised error response to a quarantine directory in staging, and promote what lands there into the fixture set.

Replaying them

For a JavaScript suite, Mock Service Worker intercepts at the network layer, so your real SDK runs unmodified and its own retry and parsing behaviour is part of what you are testing. The Node entry point is setupServer from msw/node, handlers come from http and HttpResponse in msw, and the server exposes listen, resetHandlers and close for the test lifecycle.

import { http, HttpResponse } from "msw";
import { setupServer } from "msw/node";
import { afterAll, afterEach, beforeAll, expect, it } from "vitest";
import rateLimited from "./fixtures/anthropic-429-rate-limit.json";

const server = setupServer();

beforeAll(() => server.listen());
afterEach(() => server.resetHandlers());
afterAll(() => server.close());

it("waits for retry-after and then succeeds", async () => {
  let call = 0;
  server.use(
    http.post("https://api.anthropic.com/v1/messages", () => {
      call += 1;
      if (call === 1) {
        return HttpResponse.json(rateLimited.body, {
          status: rateLimited.status,
          headers: rateLimited.headers,   // includes retry-after
        });
      }
      return HttpResponse.json({ type: "message", content: [{ type: "text", text: "ok" }] });
    }),
  );

  const result = await callModel({ input: "hello" });
  expect(call).toBe(2);
  expect(result.text).toBe("ok");
});

In Python the equivalents are the responses library for the requests stack, or a VCR.py cassette recorded in a mode that refuses to make new requests. Whichever you use, replay the headers as well as the body — a mock helper that only takes a JSON body is the exact shortcut that leaves backoff untested.

Keeping the fixtures honest

A recorded fixture is a snapshot of a moving surface, so add a job that re-provokes the cheap ones — 401, 400, 404 — against the live API on a schedule and compares status, error type and the set of header names against what is on disk. Do not compare message text: a provider is free to reword a message and a suite that fails on prose gets muted. Compare the fields your code reads, and let the job open an issue rather than fail a deploy, because a changed error shape is something to look at rather than something to block on. This is the same idea as replaying real traffic, applied to the paths that are hardest to reach on purpose.