Migrating Retry and Backoff Logic Between Providers

Retry code survives a provider migration syntactically and dies semantically. The loop still runs, the backoff curve is still correct, and it now spends thirty seconds retrying a request that was never going to succeed — while giving up immediately on one that would have.

The symptom

There are two shapes, and they look nothing alike. In the first, latency at the tail explodes: p99 goes from two seconds to forty, and the logs show the same request attempted five times with growing sleeps before failing with the same message it failed with the first time. In the second, the error rate rises with no latency change at all — transient failures that the old provider’s classifier caught are now returning straight to the caller.

Both come from the same root cause: the predicate that decides whether to retry was written against one provider’s error vocabulary and moved to another without being re-derived.

Retry is a classifier, not a curve

Almost all writing about retries is about the curve — exponential backoff, jitter, caps. The curve is the easy part and it ports perfectly, because it is arithmetic with no provider in it. The part that does not port is the function that answers “is this failure worth attempting again?”

That function has to distinguish three categories, and only the first two are commonly recognised:

• Transient — capacity, overload, a dropped connection, a gateway timeout. Retry with backoff. Time fixes it.
• Permanent — malformed request, unknown model id, bad credentials. Retrying is pure waste and will produce the identical response every time.
• Permanent wearing a transient status code — the category that breaks migrations. An exhausted billing quota that arrives as HTTP 429 is not going to clear in eight seconds, but a classifier that keys on the number will treat it exactly like a rate limit.

The third category is why the classifier must read the error body, not just the status line. Providers put the discriminator in a typed field: an error object with a type such as rate_limit_error, overloaded_error, invalid_request_error or authentication_error on one side; an error object carrying a code such as rate_limit_exceeded or insufficient_quota on the other. Those two codes share a status and have opposite retry semantics.

The four misclassifications

These are the ones that actually show up after a move, in both directions.

• Quota treated as rate limit. A 429 whose body says the account is out of credit. The retry loop burns its full budget, then surfaces the same error much later than it needed to. Fix: read the error code and treat quota exhaustion as permanent, with its own alert.
• A non-standard 5xx dropped on the floor. Not every capacity signal is a 503. At least one major API signals overload with HTTP 529, which is outside the registered range and which hand-rolled classifiers written as status === 500 || status === 503 simply miss. A range check (status >= 500) catches it; an enumeration does not.
• Payload-too-large treated as transient. A 413 is deterministic in the input. Retrying it identically is guaranteed to fail; the only useful response is to shrink the request.
• Mid-stream errors invisible to the classifier. This is the nastiest one. If you requested a stream and the provider has already sent HTTP 200 headers, a subsequent failure arrives as an event inside the stream, not as a status code. Retry logic wrapped around the HTTP call never sees it, so a failure that is perfectly retryable in principle is reported to the user as a truncated answer. The stream consumer needs its own error handling.

The double-retry trap

Both major SDKs retry internally by default. Two attempts is a common default, which means a client configured with maxRetries: 2 makes up to three HTTP requests per call. Wrap that in your own three-attempt loop and one logical request becomes nine, with the backoff curves multiplying rather than adding. Under a real rate limit you have now built an amplifier pointed at the thing that is already saturated.

Pick one layer. The usual right answer is to set the SDK’s retry count to zero and own the policy yourself, because only your layer knows about your deadline, your queue depth and your fallback provider. If you would rather keep the SDK’s retries, delete your loop — but then you also inherit the SDK’s classifier, which is generic by necessity.

Whichever you pick, honour the server’s hint. When a retry-after header is present it is better information than any curve you computed, and ignoring it in favour of your own exponential delay is how a rate limit becomes a sustained outage.

There is a third layer people forget: the load balancer or service mesh in front of your own service, which may also retry on a gateway timeout. A model request that takes ninety seconds is not unusual, and an infrastructure timeout set for ordinary web traffic will cut it off and re-issue it — producing a duplicate generation you paid for twice and, if the call had side effects, executed twice. Raise the timeout on that path and disable retries there explicitly, rather than assuming defaults tuned for a database query apply to inference.

Rebuilding the classifier

The classifier belongs beside the provider implementation, not in shared code — this is one of the things an adapter layer should deliberately not abstract. A workable shape:

type Verdict =
  | { retry: false; reason: string }
  | { retry: true; afterMs: number | null };

function classify(err: unknown): Verdict {
  // 1. Transport-level: no response was produced at all.
  if (isConnectionError(err) || isTimeout(err)) {
    return { retry: true, afterMs: null };
  }

  const status = statusOf(err);
  const code = errorCodeOf(err);   // provider-specific body field

  // 2. Permanent-in-disguise, checked BEFORE the status range.
  if (code === "insufficient_quota") {
    return { retry: false, reason: "billing" };
  }

  // 3. Rate limiting: honour the server's hint when present.
  if (status === 429) {
    return { retry: true, afterMs: retryAfterMs(err) };
  }

  // 4. Range check, not an enumeration — catches non-standard 5xx.
  if (status !== null && status >= 500) {
    return { retry: true, afterMs: retryAfterMs(err) };
  }

  // 5. Everything else in 4xx is deterministic in the request.
  return { retry: false, reason: code ?? String(status) };
}

Two habits make this durable. Log the verdict alongside the raw error code on every failure, so that the first time an unfamiliar code appears you can see how it was classified rather than inferring it from behaviour. And write the classifier’s table as tests against recorded error bodies — a fixture per code, asserting the verdict — because that is the artefact that makes the next provider move a half-day rather than a fortnight.