Designing a Retry Policy That Doesn’t Double-Charge

Standard retry advice — exponential backoff, jitter, a cap — assumes that a failed attempt cost nothing. Against a metered generative endpoint that assumption is false often enough to matter, and the cases where it is false are exactly the cases where you most want to retry.

The question standard advice skips

Before retrying, the usual question is “could this succeed if I try again?”. There is a second question here: “did the previous attempt already produce billable work?”. The two are independent, and the second one determines what a retry actually costs.

Consider three failures that a naive policy treats identically. A connection refused before the request was sent: nothing happened at the provider, the retry is free, retry immediately. A 500returned after eleven seconds: the model may well have generated a complete answer that was lost on the way back, and you were probably charged for it. A read timeout at your end while tokens were still flowing: the generation is very likely still running, to completion, on the provider’s side, and you will be billed for all of it.

Same policy, three quite different costs. A policy that ignores this can double or triple your spend during exactly the period when something is already going wrong.

Free failures and billable failures

The word “assume” is doing real work in that table. You usually cannot know whether a specific failed attempt was billed until the usage appears in an invoice or a usage endpoint. So the policy should be built on the pessimistic assumption and reconciled later, rather than the reverse.

The timeout is the dangerous one

A client-side timeout stops you waiting. It does not necessarily stop the work. If you abandon the socket without an explicit cancellation, the provider may continue generating to completion and bill you for the whole thing — you simply never see the output. Retry twice on a slow provider and you have paid three times for one answer, of which you received none.

Three mitigations, in order of importance. First, always cancel explicitly: abort the request rather than letting it lapse, so that the provider has the opportunity to stop. Second, set the timeout from a real distribution rather than a round number — a timeout below your own p99 turns normal slow requests into billable failures, and this is the single most common self-inflicted version of this problem. Third, prefer a fallback to a retry after a timeout: if the route was slow enough to time out once, the same route is the least likely to be fast the second time, and a different rung costs the same as a repeat while being more likely to work.

A policy bounded by spend

The standard bound is a maximum attempt count, which treats a hundred-token classification and a fifty-thousand-token document the same. Bound by money as well.

type RetryPolicy = {
  maxAttempts: number;
  maxSpendCents: number;      // for the whole operation, across attempts
  deadline: Deadline;
};

async function withRetries<T>(
  call: (attempt: number) => Promise<{ value: T; costCents: number }>,
  policy: RetryPolicy,
): Promise<T> {
  let spent = 0;
  let attempt = 0;

  for (;;) {
    attempt++;
    try {
      const { value, costCents } = await call(attempt);
      return value;
    } catch (raw) {
      const e = normalise(raw, currentRoute());
      spent += estimatedCostOf(e);            // billable failures cost real money

      if (!e.retryable) throw e;
      if (attempt >= policy.maxAttempts) throw new Exhausted(e, spent);
      if (spent >= policy.maxSpendCents) throw new BudgetExhausted(e, spent);

      const wait = e.retryAfterMs ?? fullJitter(attempt);
      // Never sleep past the deadline: waking up to make a call that is
      // already too late is the purest way to buy something worthless.
      if (wait + minimumViableMs > policy.deadline.remaining())
        throw new DeadlineExceeded();

      await sleep(wait);
    }
  }
}

Three of those guards are the ones normally missing. The spend bound stops an expensive request from retrying itself into a large number. The deadline check before sleeping stops the classic pattern of backing off for four seconds into a deadline that expires in two. And retryAfterMs taking precedence over your computed backoff respects information the provider gave you, which is better than anything you can infer.

Backoff, jitter and retry budgets

Two pieces of published engineering writing are worth applying directly here rather than reinventing.

AWS’s Architecture Blog article “Exponential Backoff and Jitter” (2015) compares backoff strategies under contention and makes the case for full jitter — sleeping a uniformly random duration between zero and the exponential bound, rather than the bound itself or the bound plus a small random offset. The mechanism is straightforward: synchronised clients that fail together and back off by identical amounts retry together, so the second wave is as concentrated as the first. Randomising the whole interval spreads them.

function fullJitter(attempt: number, baseMs = 500, capMs = 20_000) {
  const bound = Math.min(capMs, baseMs * 2 ** (attempt - 1));
  return Math.random() * bound;
}

Google’s SRE book describes the complementary control: a retry budget, in which a client tracks the ratio of retries to ordinary requests over a window and stops retrying when the ratio exceeds a threshold. Per-request attempt caps do not prevent a system from tripling its own load during a partial outage, because every request independently decides to retry. A budget caps the aggregate. Here it also caps aggregate spend, which is a second reason to want it.

The two compose: jitter fixes the timing of retries, the budget caps their quantity, and the circuit breaker stops them entirely once the dependency is clearly unwell.

Retrying a stream that already started

The hardest case. Two hundred tokens have been streamed to the user and the connection drops. Restarting produces a different answer, which is jarring if the user was reading, and pays for the first two hundred tokens twice.

There are three defensible responses and the right one depends on the feature. Discard and restart, telling the user, which is honest and simple and correct for short answers. Resume from a durable log if you built one, which requires that generation was decoupled from delivery — the transport page covers this — and is the only option that does not pay twice. Or keep the partial and stop, which works when a truncated answer is still useful, such as a list where the first items are the important ones.

What is not defensible is silently restarting and replacing the text the user was reading, and it is what the naive implementation does. Whatever you choose, record the partial output and its cost against the operation, because a system that only accounts for completed generations will systematically understate what it spends during a bad hour — which is precisely the hour you will later be asked about.