Testing Retry Logic Around an LLM API Call

A retry block that has never executed in a test is a retry block that does not work. It is the classic dead branch: written under pressure after an outage, never entered again, and wrong in a way that only shows up during the next outage.

The four assertions

Timing is not on this list. Whether the second attempt waited the right amount of time is a separate question with its own machinery, and it belongs to testing exponential backoff. What this page asserts is that the path runs and that it runs on the right inputs:

• The call count. Exactly two attempts for one transient failure, not one and not three.
• The second body is the same as the first. A retry that rebuilds its request can silently pick up a new timestamp, a new random id, or a conversation array that has already had the failed turn appended to it.
• The caller sees success. The retry is transparent; no partial result, no error logged as if it were fatal.
• A permanent failure is not retried. Retrying a 400 is three times the cost for three times the same error, and it is the most common bug in hand-written retry code.

Turn off the SDK’s retries first

This is the trap that makes a retry test worthless while looking perfect. Most vendor SDKs retry internally by default — the OpenAI Node client documents a default of two retries on connection errors, 408, 409, 429 and 5xx, configurable per client or per request through maxRetries in its request options.

So if you inject one 503 and your wrapper reports success, you have learned nothing: the SDK may have swallowed the failure before your code ever saw it. Your catch block never ran, your test is green, and the retry logic you shipped is still dead.

import OpenAI from "openai";

// In tests: exactly one HTTP attempt per call, so the only retries
// on the wire are the ones your code performs.
export const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  maxRetries: 0,
});

Check the value rather than assume it. Construct the client exactly the way your production code does, read the configured retry count back and assert it in a test of its own; a one-line assertion is cheap insurance against a refactor that reintroduces the default six months from now. The same reasoning applies one layer lower, where some HTTP agents transparently re-dial a connection that was reset while idle — a retry that is invisible to both your code and the SDK, and that will make an attempt-count assertion read one higher than either layer can explain.

Forcing exactly one transient failure

A one-shot handler followed by a permanent one gives you a deterministic failure sequence. In msw, the third argument to a handler carries { once: true }, which retires it after a single match:

import { http, HttpResponse } from "msw";
import { expect, it } from "vitest";
import { server } from "./setup";
import { classify } from "../src/classify";

it("retries once after a 503 and sends an identical body", async () => {
  const bodies: any[] = [];

  server.use(
    http.post(
      "https://api.openai.com/v1/chat/completions",
      async ({ request }) => {
        bodies.push(await request.json());
        return HttpResponse.json({ error: { message: "overloaded" } }, { status: 503 });
      },
      { once: true },
    ),
    http.post("https://api.openai.com/v1/chat/completions", async ({ request }) => {
      bodies.push(await request.json());
      return HttpResponse.json({
        model: "gpt-4o-mini",
        choices: [{ index: 0, finish_reason: "stop", message: { role: "assistant", content: "billing" } }],
        usage: { prompt_tokens: 30, completion_tokens: 1, total_tokens: 31 },
      });
    }),
  );

  const label = await classify("card declined");

  expect(label).toBe("billing");
  expect(bodies).toHaveLength(2);
  expect(bodies[1]).toEqual(bodies[0]);
});

The last assertion is the one that catches real bugs. A wrapper that re-runs its prompt builder between attempts, or that appends the failed assistant turn to the message array before retrying, produces a second request that differs from the first — sometimes larger every time, which is how a retry storm turns into a bill. Comparing the two captured bodies for deep equality catches every variant of that in one line.

If your seam is an injected interface rather than HTTP, the same test is shorter: a fake that fails once and then succeeds, as in the hand-written stub. Use the HTTP version when the retry lives inside the adapter and the fake version when it lives above it.

Order matters in that handler list. Handlers are matched in the order they were registered, and a one-shot handler is consumed by the first request it matches and then skipped for the rest of the test. So the { once: true } failure has to come before the permanent success handler; reverse the two and every request is answered by the success handler, the failure never happens, and you have a green test that proves nothing. That is the same class of false pass as leaving the SDK’s own retries switched on, arriving from a different direction, and neither one announces itself.

Proving what is not retried

The negative test is as important as the positive one and takes three lines. Return a permanent error and assert the attempt count is one:

it("does not retry a 400", async () => {
  let attempts = 0;
  server.use(
    http.post("https://api.openai.com/v1/chat/completions", () => {
      attempts += 1;
      return HttpResponse.json(
        { error: { message: "Invalid schema for tool 'lookup'", type: "invalid_request_error" } },
        { status: 400 },
      );
    }),
  );

  await expect(classify("…")).rejects.toThrow();
  expect(attempts).toBe(1);
});

Write one of these per status you have an opinion about. The set that usually deserves an explicit test: 400 and 422 must not retry, since the request is wrong and will stay wrong; 401 and 403 must not retry, since a credential does not become valid by asking again; 404 must not, since a model id that does not exist will not appear; 408, 409, 429 and 5xx should. A content-filter refusal that arrives as a 200 with a finish_reason of content_filter must not retry either, and that one is easy to miss because it is not an error status at all.

Attempt budgets and the cost assertion

Finally, assert the ceiling. A retry loop with an off-by-one gives you four attempts where you configured three, and every extra attempt on a long prompt is a full input charge — a failure that produced no output tokens still bills for everything you sent.

it("gives up after the configured number of attempts", async () => {
  let attempts = 0;
  server.use(
    http.post("https://api.openai.com/v1/chat/completions", () => {
      attempts += 1;
      return HttpResponse.json({ error: { message: "overloaded" } }, { status: 503 });
    }),
  );

  await expect(classify("…", { maxAttempts: 3 })).rejects.toThrow(/after 3 attempts/);
  expect(attempts).toBe(3);
});

Note that the assertion is on attempts, not on retries. Half the off-by-one bugs in this code come from the two words being used interchangeably in the same function: name the config field maxAttempts and the ambiguity disappears from the code as well as from the test. The economics of getting this wrong are worked through in what retries cost.

One more test earns its place if your retry wrapper is shared: that exhausting the attempts raises an error carrying the last underlying failure rather than replacing it. A generic Error("retries exhausted") discards the status, the request id and the provider message — which are the only three things anyone will want when they open the alert at three in the morning.