Why the Retry Branch Stopped Firing After the Migration

A multi-step workflow has a branch that decides what to retry. It was written against one provider’s error vocabulary, and after a cutover it is either dead code or an infinite loop. Both are common and they have the same root.

Two failures from one line of code

The condition usually looks something like if status in (429, 500, 502, 503): retry(), sometimes refined with a check on the provider’s error type string. After a migration it fails in one of two directions.

Dead branch. The new provider signals overload with a status the condition does not list, so a genuinely transient failure falls through to the terminal path. The workflow marks the job failed, the compensating logic runs, and the user sees an error for something that would have succeeded on the second attempt. In the logs it looks like a spike in hard failures with no corresponding spike in retries, which is the diagnostic signature.

Runaway branch. The opposite, and more expensive. A condition that treats 429 as always retryable now catches errors that are not capacity problems at all — a spend cap reached, a prepaid balance exhausted, an organisation-level usage limit — which some providers return with the same 429 status. Those never succeed on retry. A backoff loop with a generous ceiling will keep trying until it exhausts its attempts on every single request, multiplying latency and, where the failure is per-request rather than account-wide, multiplying spend.

The codes are not the same codes

Anthropic’s errors reference documents a status-to-type mapping that includes several values a branch written elsewhere will not have: 402 billing_error, 409 conflict_error, 413 request_too_large, 504 timeout_error, and — the one that most often kills a retry branch — 529 overloaded_error. A condition listing 500, 502 and 503 does not match 529, so the single most common transient failure is the one that falls through. The error body is a top-level object with type set to the string error, an inner error object carrying type and message, and a request_id.

OpenAI documents 429 as covering both rate limiting and quota exhaustion, with distinct codes underneath — its error-codes guide lists credit_balance_exhausted, organization_spend_limit_exceeded, project_spend_limit_exceeded and organization_usage_limit_exceeded alongside ordinary rate limiting, and 503 for an overloaded engine. That is the runaway case in one sentence: same status, one meaning retryable and four meaning stop.

So the two rules that follow are: never branch on status alone where the provider distinguishes underneath it, and never assume a status you have not seen in that provider’s documentation is absent. The safe default for an unrecognised 5xx is a small bounded retry; the safe default for an unrecognised 4xx is to fail immediately, because 4xx means the request as sent will not succeed however many times you send it.

The 200 that is actually an error

The failure that escapes every status-based branch is the mid-stream error. When you request a streamed response, the provider commits to an HTTP 200 and opens the event stream before it knows the request will complete. If something fails after that point, the failure arrives as an event inside a successful response. Anthropic’s errors documentation says this explicitly — error handling for a stream does not follow the standard status-code mechanism — and directs readers to the error events in the streaming format.

A workflow that wraps the call in a try block and inspects the HTTP status sees success and proceeds with whatever partial text arrived. Downstream, the symptom is a truncated answer with no error anywhere in the logs, which is the hardest possible thing to diagnose from the outside. The fix is that the stream consumer, not the HTTP layer, owns error detection: it must treat an error event as an exception, and it must treat a stream that ends without the documented terminal event as an exception too. A stream that simply stops is not a completed stream.

The related trap is a terminal reason that is not a failure but is not completion either. The Messages API can return pause_turn, which means the turn was interrupted and is expected to be continued, and model_context_window_exceeded, which is a real failure but arrives as a successful response rather than a 4xx. A branch that treats anything other than normal completion as a hard error will abort a turn that was meant to resume; a branch that treats every 200 as success will silently accept a context overflow.

Normalise into your own taxonomy

The durable fix is to stop letting provider vocabulary reach the workflow. Every adapter maps whatever it received into a small closed set that your recovery logic branches on, and the mapping table is the only place that knows about status codes and type strings.

export type Failure =
  | "transient"        // retry with backoff: overload, 5xx, timeouts, socket errors
  | "throttled"        // retry, but honour retry-after; capacity, not fault
  | "exhausted"        // account-level: spend cap, credit balance, usage limit — do NOT retry
  | "invalid_request"  // your payload is wrong; retrying is pointless — alert a human
  | "auth"             // key or permission; page whoever owns credentials
  | "too_large"        // reduce input and retry once, then fail
  | "content"          // refusal or filter; a different fallback, not a retry
  | "unknown";         // conservatively: one bounded retry if 5xx, else fail

// The mapping table is per provider and is the only provider-aware code.
// Everything downstream branches on Failure and never on a status code.

Two properties make this worth the indirection. Each category has exactly one correct action, so the workflow’s branch becomes a switch with no judgement in it. And exhausted being a separate category from throttled is what prevents the runaway loop, because it is the one distinction a status code cannot express. Keep the raw provider values on the log line beside the normalised one — the normalisation is for control flow, the raw string is for the support ticket.

The content category deserves its own path rather than a retry. Retrying an identical request that was refused produces another refusal and burns the budget twice; the correct response is a different prompt, a different model, or a user-facing message, which is the subject of refusal fallback logic.

Testing it without an outage

None of this can be verified in production, so inject the faults. Put a test double in front of the adapter that can return, on command, each documented status and type string for each provider you support, plus a stream that emits an error event after three chunks and a stream that simply stops. Assert on the normalised category and the action taken, not on log text.

1. Enumerate, per provider, every status and type string in its published error reference. This list is small and it is the input to the mapping table.
2. Write one test per entry asserting the normalised category. A new provider is then a table plus a set of expectations, not a code change.
3. Add the two stream cases explicitly, since they cannot be reached through a status code.
4. Add a budget assertion: for a request that maps to exhausted, the total number of upstream calls must be exactly one. This is the regression test for the runaway loop, and it is the one that pays for itself.
5. Re-run the whole table when you upgrade an SDK major. Retry behaviour is frequently built into the client — Anthropic’s SDKs retry transient failures twice by default and honour retry-after — so your attempt count is the product of two retry layers, and a default change silently multiplies it.