Cost of Retries, Failovers and Timeouts

Nobody models retries, and for most failure modes that is fine — the adjustment is a couple of percent. There is one failure mode where it is not fine at all, and it is the one that grows precisely when your system is already under stress.

The retries that are nearly free

When a request fails before the model generated anything — a connection refused, a 429 rate limit, a 503, a 400 for a malformed body — no tokens were produced and nothing was billed. The retry is a fresh, fully-priced request, but the failed attempt cost nothing, so the multiplier is small.

With a per-attempt failure probability f and up to n retries, the expected number of billed attempts per logical request is a geometric sum:

E[attempts] = SUM(i=0..n) f^i  =  (1 - f^(n+1)) / (1 - f)

f = 0.05, n = 3:
  E = (1 - 0.05^4) / 0.95 = 1.0526

So a 5% pre-generation failure rate with three retries adds
5.3% to the bill. As n -> infinity the ceiling is 1/(1-f) = 1.053.

The ceiling is the useful part: with a small f, adding more retries barely changes the cost, which means retry count is rarely the thing to economise on. Even a horrible f = 0.2 caps at 1/(1−0.2) = 1.25, a 25% surcharge. Uncomfortable, but not a crisis, and the real problem at that failure rate is not the money.

The expensive kind: you were billed anyway

Now the case that behaves differently. Your client times out at 20 seconds. The provider did not fail — it generated the whole answer in 26 seconds and billed you for every token. Your client has already given up, discarded the connection, and issued a second request, which also completes and is also billed. You paid twice and received once.

E[cost] = C * (1 + t)      per logical request

  t  fraction of requests that exceed the client timeout
     but complete server-side

t = 0.10  ->  10% surcharge, and no failure appears in
              your error rate at the provider's end

Three things make this worse than the arithmetic suggests, and all three are correlated with each other.

• The timeout is usually set near p95. Which means t is 5% by construction, not by accident. Setting a timeout at the p95 of your own latency distribution guarantees you pay double on one request in twenty.
• It gets worse exactly when load does. Latency rises under load, so t rises under load, so retries rise, so load rises. This is a retry storm and its cost signature is a bill that spikes without a traffic spike to match.
• Long outputs are the ones that time out. Generation time scales with output tokens, so the requests that exceed the timeout are systematically the expensive ones. The doubled requests are not a random sample; they are drawn from the tail of your cost distribution.

The same logic applies to a client that aborts a streamed response. Whether you stop being billed at the abort depends on whether the cancellation propagates to the provider and whether the provider stops generating — which is worth testing rather than assuming.

Failover has a different price on each leg

Retrying the same provider costs C again. Failing over to a different provider costs whatever that provider charges, and the two can differ by a lot — that is frequently the reason the second provider is the fallback rather than the primary.

E[cost] = C1 + f * C2               pre-generation failure:
                                    leg 1 billed nothing
E[cost] = C1 + t * (C1 + C2)        timeout after generation:
                                    leg 1 billed in full

With C1 = $0.004, C2 = $0.011, f = 0.03:
  E = 0.004 + 0.03*0.011 = $0.00433     (+8%)

With the same rates but a post-generation timeout t = 0.03:
  E = 0.004 + 0.03*(0.004 + 0.011) = $0.00445  (+11%)

Two consequences for how a failover chain should be ordered. Put the cheaper acceptable provider first when quality is equivalent, because the first leg is paid on every request and the second only on f. And keep the chain short: each additional fallback multiplies into the tail, and a three-deep chain during an incident can bill three times for one answer.

A budget per logical request

The structural fix is to stop thinking in requests and start thinking in logical requests — one user-visible operation, which may involve several provider calls. Give each one an id, and give it a budget in money rather than in attempts.

ctx = { id: uuid(), spent_micro: 0, budget_micro: 15000 }

for attempt in 1..n:
    ceiling = estimate_ceiling(request)     # input + max_tokens
    if ctx.spent_micro + ceiling > ctx.budget_micro:
        raise BudgetExceeded(ctx)
    resp = call(provider, request, idempotency_key=ctx.id)
    ctx.spent_micro += resp.cost_micro      # actual, not estimate
    if ok(resp): return resp

Three details in that sketch carry the weight. The ceiling check happens before the call, because cost is only known afterwards. The accumulator uses the actual cost once it is known, so estimation error does not compound. And the idempotency key is the logical request id, so a retry that the provider recognises as a duplicate can be served from its record rather than regenerated — where the provider supports it, that turns the timeout case from double billing into single billing, which is the single highest-value line in this page.

Policy choices that follow from the maths

• Set the timeout from the output budget, not from a round number. Expected generation time is roughly ttft + max_tokens / rate. A timeout below that is a timeout you have chosen to pay for. If you want a 20-second bound, bound max_tokens to match it.
• Never retry a 400. A malformed request will be malformed again. Retrying client errors is free in tokens and expensive in the rate limit you burn.
• Always retry a 429 with backoff and jitter. No tokens were billed, and synchronised retries are what turn a brief rate limit into an outage.
• Retry timeouts at most once, and prefer a cheaper model. The second attempt is the one you are paying double for; making it cheaper halves the penalty.
• Count retries as a metric, separately from errors. Attempts-per-logical-request is the number that tells you whether your bill is being inflated by the retry layer, and it does not appear in any error dashboard.
• Cap the total, not the depth. A budget in micro-dollars survives a change to the retry policy, a change to the model, and a change to the fallback chain. A hard-coded “three attempts” survives none of them.