What a Provider Outage During a Migration Window Means for Your Rollback

You are at fifty percent on the new provider when it starts degrading. The dashboard looks nearly fine, the rollback trigger has not fired, and you have about ten minutes to decide something. This is the case the plan should have assumed rather than treated as bad luck.

What a partial outage looks like from your side

Total outages are easy and rare. The common shape is partial: a fraction of requests failing with a capacity error, elevated latency on the rest, and everything recovering within the hour. Providers give this its own status code — Anthropic returns 529 with error type overloaded_error when the API is temporarily overloaded, distinct from 500 api_error for an internal fault, 504 timeout_error for a request that timed out in processing, and 429 rate_limit_error for your own account’s limits. Anthropic’s error documentation enumerates the full set. Those four mean very different things during a cutover and it is worth having them separated in your metrics before you need to read them under pressure.

There is a fifth shape that catches people out: an error arriving after a 200. On a streaming response the status is committed before the body is produced, so a mid-stream failure arrives as an error event inside the SSE stream. Anything counting HTTP statuses will record that request as a success. If your cutover is on a streaming path, this is the failure mode most likely to be invisible.

Why the error-rate trigger does not fire

Almost every cutover plan carries a trigger of the form “roll back if the error rate on the new provider exceeds two percent”. During the exact event that trigger was written for, it stays quiet, and the reason is in your own client.

Official SDKs retry transient failures automatically. Anthropic’s documentation states its SDKs retry connection errors, rate limits and 5xx server errors with exponential backoff, twice by default, honouring the retry-after header when present; other vendors’ clients behave similarly. So a request that hits a 529, backs off, and succeeds on the second attempt is recorded by your application as one successful request that took three seconds longer than usual. It contributes nothing to the error rate.

A capacity event that fails a third of attempts therefore shows up as almost no errors and a large latency shift. If you are at fifty percent traffic split, that latency shift is further halved in any metric averaged across both arms, and if your dashboard shows the mean rather than a high percentile it may be barely visible at all. The trigger is not badly chosen; it is measuring a quantity the retry layer has already erased.

There is a second-order effect worth knowing about during a ramp. Backoff plus a queue means retries pile up behind the slow requests, so a brief capacity event can exhaust a connection pool or a worker pool and take out requests that would have succeeded — including, if the pools are shared, requests routed to the old provider. That is how a partial outage on the arm you are testing becomes an incident on the arm you are not.

A trigger that sees it

Write the trigger against quantities the retry layer cannot hide, and measure each arm separately.

• Latency at a high percentile, per arm. p95 or p99 of end-to-end time including retries, on the new arm alone, compared against the old arm in the same window rather than against a historical baseline. Comparing arms controls for everything that is happening to both.
• Retry count as a first-class metric. Have the client emit attempts per logical request. This is the single most sensitive indicator of provider trouble and most teams do not collect it. It moves before latency does.
• Errors counted before retries, not after. Instrument at the HTTP layer as well as the application layer, so you can see the 529s that were subsequently retried away.
• Stream completion rate. The fraction of streams that reached a terminal event, separately from HTTP status, which is the only thing that catches the post-200 failure.
• An absolute time bound on the decision. “If any of the above is anomalous for more than N minutes, roll back”. Without a bound, a degradation that never quite crosses a threshold occupies the team indefinitely, which is its own outage.

Roll back, hold, or continue

Three options, and the decision should be made from the plan rather than from the mood in the channel.

Roll back when users are affected now. This is always available and always correct under user impact. Do not weigh it against the schedule; a cutover deferred by a week costs a week, and a bad hour in production costs more than that.

Hold — freeze the ramp at its current percentage without reducing it — when the degradation is contained and your fallback is absorbing it. Holding is underused and is often the right answer, because rolling back to zero discards the evidence the ramp has produced and means starting the whole sequence again. A hold has to have a deadline attached at the moment it is declared, or it becomes a decision nobody made.

Continue only if the provider has confirmed the event is resolving and your own metrics agree. A status page that has not updated is not confirmation, and your own metrics lead a status page by a comfortable margin most of the time.

Whichever you pick, record the timestamp, the metric values that drove it and who made it, while it is happening. The migration report needs it and nobody reconstructs it accurately afterwards.

The rollback path must not depend on the target

The failure that turns an inconvenience into an incident is a rollback mechanism that needs the thing that is broken.

If the traffic split lives in a config service, the rollback needs that service to be up. If it needs a deploy, the rollback takes as long as your pipeline, which during an incident is longer than usual. If the new provider is also serving a classifier that decides routing, the rollback path runs through the outage. Test the rollback before the cutover, from the state you would actually be in — not by reverting a commit in a quiet moment, but by flipping the flag under load and watching traffic move.

Two properties make a rollback dependable. It should be a configuration change that takes effect in seconds without a deploy, and it should be executable by whoever is on call at three in the morning without needing the person who designed the migration. If either is untrue, fix that before starting the ramp, because you will not fix it during one.

Finally, assume this happens rather than treating it as a rare edge case. Over a multi-week cutover window, a capacity event somewhere in the chain is close to expected, which is why circuit breakers and a tested failover path belong in place before the migration starts and not as a follow-up. Move your uptime monitoring onto the new provider before the traffic, not after: monitoring that only watches the old arm is blind during exactly the window it exists for.