Testing Idempotency of a Retried LLM Request

Every retry rests on an assumption that the failed attempt did nothing. The expensive bug in this cluster is the case where that is false: the request arrived, the work happened, and only the response was lost.

The failure has a specific shape

It is worth being precise, because the general phrase “network error” hides two very different events. In one, the request never reached the server — connection refused, DNS failure, TLS handshake rejected — and a retry is free. In the other, the request was received and processed, and the failure happened on the way back: a read timeout, a dropped connection mid-response, a load balancer returning 502 after the upstream had already committed.

From inside your process these are indistinguishable. Both surface as an exception with no response body. That is the entire problem, and it is why the fix cannot be “detect which one happened” and has to be “make the second attempt safe either way”.

For a plain completion the damage is bounded: you pay twice and get two answers, one of which you discard. Annoying, and it adds up, but not corrupting. The damage becomes real when the request had a side effect, and in an LLM system it very often did — because the model asked for one.

Retry budgets make the exposure worse rather than better, which is counter-intuitive enough to be worth stating. Three attempts against an endpoint that already succeeded means up to three units of downstream work, and the longer your backoff the more likely it is that the first attempt has finished by the time the second arrives. So the configuration that looks most conservative — patient waits, generous retries — is also the one most likely to produce a duplicate.

Reproducing it deterministically

The test needs a fake provider that records what it processed independently of what it returned. Then the handler can succeed internally and fail on the way out:

import { http, HttpResponse } from "msw";
import { expect, it } from "vitest";
import { server } from "./setup";
import { generateAndSend } from "../src/generate";

it("does not bill twice when the first response is lost in transit", async () => {
  const processed: string[] = [];   // what the "provider" actually did

  server.use(
    http.post(
      "https://api.openai.com/v1/chat/completions",
      async ({ request }) => {
        processed.push(request.headers.get("idempotency-key") ?? "(none)");
        // Work completed server-side, then the response is lost.
        return HttpResponse.error();
      },
      { once: true },
    ),
    http.post("https://api.openai.com/v1/chat/completions", async ({ request }) => {
      const key = request.headers.get("idempotency-key") ?? "(none)";
      if (processed.includes(key)) {
        // A real idempotent endpoint replays the stored response.
        return HttpResponse.json(storedReply, { headers: { "idempotent-replay": "true" } });
      }
      processed.push(key);
      return HttpResponse.json(storedReply);
    }),
  );

  await generateAndSend("draft the renewal email");

  expect(processed).toHaveLength(2);
  expect(new Set(processed).size).toBe(1);        // the SAME key both times
  expect(processed[0]).not.toBe("(none)");
});

HttpResponse.error() produces a network-level failure rather than an HTTP status, which is exactly the ambiguous case: your client sees a connection error and cannot tell that the server did the work. The two assertions at the end are the page in miniature — two attempts reached the server, and both carried one key.

Notice what the test never does: it does not compare the two responses. The properties under test are a count and an identity, and both of those are yours. A test that asserted the second completion matched the first would fail whenever the sampler produced different words for the same prompt, which is most of the time — and the person who inherits it would learn to ignore it rather than to read it.

The key must not be generated per attempt

Here is the bug this test is designed to catch, and it is subtle enough to survive review:

// WRONG — a new key on every attempt makes the header decorative
async function callWithRetry(body) {
  for (let attempt = 1; attempt <= 3; attempt++) {
    try {
      return await client.chat.completions.create(body, {
        idempotencyKey: crypto.randomUUID(),   // regenerated inside the loop
      });
    } catch (e) { if (!transient(e)) throw e; }
  }
}

// RIGHT — one key per unit of work, reused by every attempt of it
async function callWithRetry(body, { workId }) {
  const idempotencyKey = `renewal-email:${workId}`;
  for (let attempt = 1; attempt <= 3; attempt++) {
    try {
      return await client.chat.completions.create(body, { idempotencyKey });
    } catch (e) { if (!transient(e)) throw e; }
  }
}

The OpenAI Node client accepts idempotencyKey among its per-request options and sends it as an Idempotency-Key header; it also generates one itself so that its own internal retries reuse a single key. That last detail is the reason the wrong version above looks fine in a smoke test: the SDK’s retries are safe, and only your loop reintroduces the duplicate.

Derive the key from the unit of work, not from the attempt and not from the request body. Hashing the body seems clever and is wrong in both directions: two genuinely separate requests with identical content get collapsed into one, and a request that includes a timestamp gets a new key on every retry. A stable business identifier — the job id, the message id, the invoice id — is what you want, and it should be the same identifier that appears in your logs.

The duplicate that actually hurts

Now the part most treatments of this topic miss. In an agentic system, the request that must not be duplicated is usually not the completion at all. It is the tool the model asked you to run: issuing a refund, sending an email, creating a ticket, posting to a webhook. The provider’s idempotency key does nothing for that, because your executor is on your side of the boundary.

Which means the assertion belongs on the side effect, not on the header:

it("executes a tool call exactly once across a retried turn", async () => {
  const refunds: Array<{ orderId: string }> = [];
  const tools = {
    async refund(args: { orderId: string }, ctx: { callId: string }) {
      if (executed.has(ctx.callId)) return executed.get(ctx.callId);   // dedupe
      const result = { ok: true };
      refunds.push(args);
      executed.set(ctx.callId, result);
      return result;
    },
  };

  // The model turn is retried after the tool ran but before the follow-up
  // completion came back — the classic mid-loop failure.
  await runAgentTurn({ tools, transport: failAfterFirstToolCall() });

  expect(refunds).toHaveLength(1);
});

The dedupe key here is the tool call id the provider assigns to each entry in tool_calls, which is stable within a conversation and is the natural identity for “this specific requested action”. Two things follow that are worth their own assertions: a replayed call must return the same result as the first, not merely skip — the model reads that result and will behave differently if it changes; and the dedupe record has to outlive the process, because the retry that matters most is the one after a deployment restarted the pod mid-loop.

What the key does not protect

Write one final test that documents the boundary, so nobody over-trusts the mechanism:

• Keys expire. Provider-side idempotency records are retained for a bounded window, typically measured in hours. A retry from a dead-letter queue drained the next morning is a fresh request no matter what key it carries.
• A key with a different body is an error, not a replay. Reusing a key while changing the prompt is a bug in your code, and the good behaviour is to fail loudly. Assert that your wrapper does not silently accept it.
• It says nothing about determinism. Idempotency means the second call has no additional effect; it does not mean two distinct calls with the same prompt return the same text. Conflating them leads to the mistake this whole cluster exists to prevent — asserting on model prose.
• Streaming complicates it. A stream that failed halfway has already delivered tokens to the user. Replaying the stored response means either re-sending from the beginning or reconciling with what was shown, and both need a deliberate decision recorded in a test.

Because of the first two, the durable place for the guarantee is your own storage rather than the provider’s. Write the unit of work and its outcome to a table with a unique constraint on the work id before you call anything, and the question stops being “did the provider deduplicate this” and becomes “have I already done this” — which you can answer, and which a test can assert on directly by counting rows.