Retries and Backoff for LLM APIs

Retrying an HTTP call is a solved problem with a well-known shape: exponential backoff, jitter, a cap, a budget. Retrying a model call adds one property no ordinary API has — the failed attempt may have cost you real money and may still be running — and that changes which failures are worth retrying at all.

Why this is not a generic retry

Three properties set LLM calls apart from the RPCs retry libraries were designed for.

• Attempts are expensive and asymmetric. A retried database read costs microseconds. A retried 30-second generation over a 40k-token prompt costs the prompt again, plus however many output tokens the first attempt produced before you gave up.
• A client timeout does not cancel the server. This is the double-billing hazard. You abort at 30 seconds, the backend keeps generating, you retry, and now two generations exist for one answer. Both are billable. The first one’s tokens go nowhere.
• Failures are frequently capacity, not faults. A 429 or an overloaded 503 means the system is already at its limit. Retrying quickly is not a neutral act — it is adding load to a system that just told you it has none spare.

The practical rule that follows: retry aggressively on failures that cost nothing, cautiously on failures that may have cost you a generation, and never on failures that will fail identically.

Which failures are retryable

Full jitter, and why not the other kinds

Plain exponential backoff synchronises clients. If a hundred callers fail at the same moment — which is what happens when a backend blips — they all wait 1 s, all retry together, all fail together, then all wait 2 s. The retries arrive as a spike shaped exactly like the one that caused the failure.

Jitter breaks the synchronisation, and the variants are not equivalent. Marc Brooker’s analysis on the AWS Architecture Blog (“Exponential Backoff And Jitter”, 2015) compares them by simulation and finds full jitter — sampling uniformly across the whole interval rather than adding a small perturbation to the top of it — the best combination of low total work and low completion time:

cap  = 20_000 ms
base =    500 ms

exponential        sleep = min(cap, base * 2**n)                   # synchronised
"equal jitter"     sleep = t/2 + rand(0, t/2),  t = min(cap, base*2**n)
FULL JITTER        sleep = rand(0, min(cap, base * 2**n))          # use this
decorrelated       sleep = min(cap, rand(base, prev * 3))

n = 0  -> full jitter samples uniformly from [0,   500] ms
n = 1  ->                                    [0,  1000]
n = 2  ->                                    [0,  2000]
n = 3  ->                                    [0,  4000]

Full jitter has a property people find uncomfortable: a retry can fire almost immediately. That is not a bug — the expected wait still doubles each attempt, and the near-zero draws are exactly what spreads a thundering herd across the interval instead of stacking it at the end.

The implementation

const BASE_MS = 500;
const CAP_MS = 20_000;

/** RFC 9110: Retry-After is either delta-seconds or an HTTP-date. Both occur. */
function retryAfterMs(res: Response): number | null {
  const h = res.headers.get("retry-after");
  if (!h) return null;

  const secs = Number(h);
  if (Number.isFinite(secs)) return Math.max(0, secs * 1000);

  const when = Date.parse(h);
  return Number.isNaN(when) ? null : Math.max(0, when - Date.now());
}

function fullJitter(attempt: number): number {
  return Math.random() * Math.min(CAP_MS, BASE_MS * 2 ** attempt);
}

type Outcome = "retry" | "stop";

function classify(status: number): Outcome {
  if (status === 408 || status === 409 || status === 429) return "retry";
  if (status >= 500) return "retry";
  return "stop";                       // every 4xx except those three
}

export async function callWithRetry(
  url: string,
  init: RequestInit,
  opts: { attempts: number; deadline: number; idempotencyKey: string },
): Promise<Response> {
  let last: unknown;

  for (let attempt = 0; attempt < opts.attempts; attempt++) {
    const left = opts.deadline - Date.now();
    if (left <= 0) break;              // no budget: do not start what nobody awaits

    try {
      const res = await fetch(url, {
        ...init,
        headers: {
          ...init.headers,
          // Lets the server collapse a duplicate attempt into the original
          // instead of generating (and charging for) it twice. Same key for
          // every attempt of the SAME logical request.
          "idempotency-key": opts.idempotencyKey,
        },
        signal: AbortSignal.timeout(Math.min(left, 60_000)),
      });

      if (res.ok) return res;
      if (classify(res.status) === "stop") return res;   // caller handles 4xx

      last = new Error("HTTP " + res.status);
      // The server's own instruction wins over our schedule. Only fall back
      // to jitter when it did not send one.
      const wait = retryAfterMs(res) ?? fullJitter(attempt);
      if (Date.now() + wait >= opts.deadline) break;
      await sleep(wait);
    } catch (err) {
      last = err;
      if (err instanceof Error && err.name === "AbortError") {
        // Timed out. The generation may still be running and billing.
        // Retry only if the budget genuinely allows another full attempt.
        if (opts.deadline - Date.now() < 2 * BASE_MS) break;
      }
      const wait = fullJitter(attempt);
      if (Date.now() + wait >= opts.deadline) break;
      await sleep(wait);
    }
  }
  throw last ?? new Error("retries exhausted");
}

const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));

Four things in there are the difference between this and a three-line retry wrapper. Retry-After is parsed in both of its legal forms and takes precedence over the local schedule. The deadline is checked before sleeping, so the code never waits four seconds to make a call whose caller gave up three seconds ago. The idempotency key is stable across attempts of one logical request and different between requests — that is what allows a server that supports it to deduplicate rather than regenerate. And a 400 is returned, not retried.

Retry budgets and the metastable failure

Per-request retry limits do not bound system-wide retry load. If every request may retry three times, a broad failure triples your offered load at precisely the moment the dependency is struggling — and a system can stay in that state after the original trigger is gone, because the retries are now the load. This is the metastable failure pattern, and it is why capped attempts are not sufficient.

The standard control is a global retry budget: a token bucket that all retries draw from, sized as a fraction of the success rate.

// Retries may add at most 10% to the offered load, process-wide.
class RetryBudget {
  private tokens = 0;
  constructor(private ratio = 0.1, private max = 100) {}

  onSuccess() { this.tokens = Math.min(this.max, this.tokens + this.ratio); }

  tryConsume(): boolean {
    if (this.tokens < 1) return false;   // budget spent: fail fast instead
    this.tokens -= 1;
    return true;
  }
}

When the dependency is healthy, successes keep the bucket full and retries always run. When it is failing, successes stop, the bucket drains, and retries stop with it — the system degrades to fast failures instead of amplifying the outage. That behaviour is the whole point, and it is the piece missing from almost every hand-rolled retry helper.

Two adjacent techniques are worth distinguishing from retrying, because they are frequently the better answer. Hedging issues a second request after a delay set at roughly the p95, and takes whichever returns first. It attacks tail latency rather than failure, and for a model call it is expensive by construction — you are deliberately paying for two generations — so it belongs only on short, cheap, latency-critical calls, and only with a hedge budget capped at a few per cent of traffic. Failing over sends the retry to a different provider entirely, which is the only variant whose second attempt does not depend on the thing that just failed.

Streaming deserves its own rule. There is no resume: if a stream dies at token 400, the retry starts from token 0 and you pay for both. That makes mid-stream retry the most expensive operation in this whole page, and it is worth asking whether the partial output is usable instead — for a chat interface, showing what arrived and offering to continue is both cheaper and more honest than silently regenerating. Where the output is structured and must be complete, retry, but count it, because a rising mid-stream retry rate is a latency problem wearing a reliability costume.

Pair all of this with a circuit breaker per endpoint, and consider whether the right response to a sustained failure is a retry at all: for a model call, failing over to a different provider is frequently a better use of the same wait, since the second attempt then does not depend on the thing that just failed. The queueing and shedding side of that is in handling 429s.