Migrating a Client's Exponential Backoff Parameters Between Providers

Your retry helper has base=1.0, factor=2, max_retries=5, and those values were tuned against a provider you no longer use. Copying them over is the default action and it produces two distinct failures: retries that give up before the new provider’s limits reset, and retries that multiply against the SDK’s own.

The bug the migration introduces

Deal with the multiplication first, because it is silent and it is almost universal. Most official SDKs retry on your behalf. The Anthropic SDKs, for example, automatically retry connection errors and HTTP 408, 409, 429 and 5xx responses with exponential backoff, with max_retries defaulting to 2 (platform.claude.com, errors). If your own helper wraps that call with five attempts, one logical request becomes up to fifteen HTTP requests, each of them consuming quota from the very limit you are backing off from.

The timeout arithmetic compounds it. A client timeout applies per attempt, so the worst-case wall clock for a single SDK call is roughly timeout × (max_retries + 1) before your own layer has retried once. With a ten-minute default timeout and both layers active, a single logical request can occupy a worker for the better part of an hour while your dashboard reports it as one in-flight call.

Pick one layer. The clean choice is to disable the SDK’s retries (max_retries=0) and own the policy, because you are the one who knows about queueing, deadlines and per-tenant fairness. The lazy but acceptable choice is to keep the SDK’s retries and delete yours. What is not acceptable is both, and the migration is the moment to check, because the two SDKs may have different defaults.

Read the provider’s signals, do not guess

Exponential backoff exists to find a delay when you do not know one. When the provider tells you the delay, guessing is strictly worse. Every major provider returns rate-limit metadata on the response headers, and the identifiers differ between them — which is precisely the part a migration has to re-map.

Anthropic documents a retry-after header on a 429 giving the seconds to wait, alongside x-ratelimit-limit-* and x-ratelimit-remaining-* headers exposing your limits and remaining quota (platform.claude.com, rate limits). Do not assume another provider spells them the same way; read the target’s reference page and write the mapping into an adapter, rather than scattering header names through your retry code.

# adapter.py — one place that knows a provider's header spelling
def retry_delay(headers) -> float | None:
    """Seconds the provider asked us to wait, or None if it did not say."""
    for name in ("retry-after", "retry-after-ms", "x-ratelimit-reset-requests"):
        raw = headers.get(name)
        if raw is None:
            continue
        try:
            v = float(raw)
        except ValueError:
            return _parse_http_date(raw)          # some providers send a date
        return v / 1000 if name.endswith("-ms") else v
    return None

Two limit classes, two recovery shapes

The reason a single backoff curve fits badly is that a 429 can mean two quite different things, and they recover on different timescales.

Request-rate limits are typically enforced over a short rolling window. If you exceed requests per minute, capacity returns within that window — often in single-digit seconds. Aggressive early retry is correct here; a first delay of thirty seconds wastes most of the recovery.

Token-throughput limits behave differently. Input and output token budgets are consumed by the size of your requests, not their count, so a single very large request can exhaust the window and the recovery is proportional to the window rather than to your retry cadence. And daily token budgets do not recover on a retry timescale at all — retrying into a spent daily limit is pure waste, and the only correct responses are to shed load, degrade to a cheaper model, or fail fast with a clear message.

Read which class you hit from the remaining-quota headers rather than inferring it from the status code, then branch. Retrying a daily-limit 429 on the same schedule as a per-minute one is the most common reason a migrated client burns its quota faster than the old one did.

Deriving the parameters

1. Find the shortest recovery window. Read the target provider’s reference for how its request-rate limit is enforced. Set base to roughly a tenth of that window — for a one-minute window, a base of five to six seconds. A base of one second against a per-minute window means five wasted attempts before the first plausible one.
2. Set the cap from your deadline, not from a habit. If the caller is an interactive request with a fifteen-second budget, the cap is the budget minus elapsed time and the honest answer is often zero retries. Cap and attempt count are properties of the caller; derive them per call site rather than globally.
3. Choose attempts so the total fits the deadline. With full jitter the expected total delay across n attempts is about base × (2^n − 1) / 2. Solve for n against your budget instead of picking five because five is a round number.
4. Use full jitter. Sleep a uniform random value in [0, min(cap, base × 2^attempt)]. Deterministic backoff synchronises every client that failed in the same instant and reproduces the thundering herd on each retry round.
5. Let the header override all of it. When the provider states a delay, use that value — clamped to your cap — and use the computed curve only when it is silent.

The implementation

import random, time

class Budget:
    """Deadline-aware retry policy. One layer only — SDK retries are off."""
    def __init__(self, base: float, cap: float, deadline: float):
        self.base, self.cap, self.deadline = base, cap, deadline

def call_with_retry(fn, policy: Budget):
    started, attempt = time.monotonic(), 0
    while True:
        try:
            return fn()
        except RateLimited as e:
            if e.limit_class == "daily":
                raise                      # retrying a spent daily budget is waste
            hinted = retry_delay(e.headers)
        except (Overloaded, ServerError, Timeout) as e:
            hinted = retry_delay(getattr(e, "headers", {}))
        except BadRequest:
            raise                          # 4xx other than 429 is never retryable

        computed = min(policy.cap, policy.base * (2 ** attempt))
        delay = hinted if hinted is not None else random.uniform(0, computed)
        delay = min(delay, policy.cap)

        elapsed = time.monotonic() - started
        if elapsed + delay > policy.deadline:
            raise DeadlineExceeded(f"gave up after {attempt + 1} attempts")

        time.sleep(delay)
        attempt += 1

Four details carry the weight. Non-429 client errors are never retried, because a malformed request will be malformed on the fifth attempt too. Daily-class limits raise immediately rather than sleeping. The deadline is checked before sleeping, so the caller fails at the deadline rather than well past it. And overload responses — a 529 on Anthropic’s API, a 503 elsewhere — go down the same path as 429 rather than being treated as fatal, because they are transient by definition.

Whatever you build, instrument it. Log the attempt number, the observed limit class and whether the delay came from a header or from the curve. Without that you cannot tell an under-provisioned quota from a badly tuned curve, and the two have opposite fixes — one is a quota increase request, the other is a config change.