What Changes About On-Call Runbooks After a Provider Migration

A runbook is a set of instructions written for someone who is tired and under pressure. Most of it survives a provider migration untouched. The part that does not is the part that names a field, a status code or a dashboard — and that is exactly the part somebody follows literally at 3am.

Which steps are actually provider-coupled

Do not rewrite the runbook. Read it once and mark every step that mentions any of five things: an HTTP status code, a field inside an error body, a response header, a vendor status page, or a support escalation path. Everything else — check the queue depth, roll back the last deploy, confirm the incident channel — is about your system and does not care who serves inference.

In practice this marks something like a fifth of the document, and it clusters in one place: the triage tree near the top, where the on-call decides whether this is a bad deploy, a bad prompt, or the provider having a day. That tree is the whole value of the runbook and it is the part most tightly bound to one vendor’s failure vocabulary.

The error envelope is not the same shape

The most common silent break is a triage step that says “look at the error code in the response body”. Provider error envelopes are not interchangeable. An OpenAI-shaped error body nests a message, a type, a param and a code under a top-level error object. Anthropic’s documented error body is a different shape: a top-level type of error, an inner error object carrying type and message, and a sibling request_id.

The consequence is specific. A log query keyed on the code field returns nothing after the migration, not because there are no errors but because the field does not exist; the classification the on-call needs lives in error.type instead. The same applies to the identifier you paste into a support ticket, which is carried in a response header whose name differs between providers — capture it from a real response and put the actual header name in the runbook rather than the one you remember.

Status codes diverge in a way that matters for routing. Anthropic documents 529 with an overloaded_error type as distinct from 500 api_error, and both as retryable with backoff. Anthropic’s errors reference lists the full set. A triage branch that reads “any 5xx means escalate to the vendor” is wrong in one direction; one that reads “500 means escalate, anything else retry” is wrong in the other. Rate limiting is the one place both families agree on a signal worth naming: a 429 with a retry-after header. The per-window limit and remaining-quota headers are prefixed differently by provider, so name them in the runbook only after reading them off a captured response.

Two failures that are not incidents

A migrated runbook usually inherits a step that treats an empty or truncated response as an outage symptom. Two non-outage causes will now trip it more often than before, and both should be diagnosed before the on-call starts looking at infrastructure.

• A policy decline is a successful request. It arrives as HTTP 200 with a terminal reason that says the model declined — a stop_reason of refusal in the Messages API, or a finish_reason of content_filter in an OpenAI-shaped one. Nothing is down. The runbook step is “check the terminal reason before paging anyone”, and the code change is covered in migrating refusal-handling logic.
• A truncation is a budget, not a failure. A response that stops because it hit the output cap reports that fact in the same terminal field. If the migration also changed the tokenizer, the same prompt and the same cap now truncate at a different point, so a previously-unreachable limit becomes reachable.

Rebuilding the coupled steps

1. In a staging project on the new provider, deliberately produce one example of each failure class the runbook branches on: an auth failure, a rate limit, an overload or 5xx if you can provoke one, a request too large, a policy decline, and a truncation.
2. Log each response in full — status line, every header, the complete body — and paste the real payloads into the runbook as fenced blocks. A runbook that shows the shape the on-call will actually see beats one that describes it.
3. Rewrite each marked step against the captured payload, changing only the field paths and status codes. Resist restructuring the tree at the same time; two changes at once means neither gets reviewed.
4. Add one field to your request log if it is not there already: which provider and which model string served this call. Every step below it in the tree depends on knowing that, and during a dual-run it is the first question anyone asks.
5. Update the known-error references: the new provider’s status page URL, the support channel, and whatever internal incident tag you use to find previous occurrences. Old incident links stay — they are still history — but mark them with the provider they belong to.

The runbook during the cutover window

While traffic is split, the runbook has to answer a question it never had to answer before: is this failing for everyone or only for the cohort on the new provider? Add a single step near the top of the tree that says how to find out — typically a saved query filtered on the provider field you just added — and put it before any step that suggests a rollback. Without it the standard reflex, roll back the last deploy, gets applied to an incident the deploy did not cause.

Keep the old provider’s branch in the document until the old binding is actually gone, then delete it in one commit rather than letting it fade. A runbook with two sets of instructions and no indication of which is live is worse than either one alone. The completion signal is the same one used in the model reference doc: when no entry has two provider bindings, the migration branch of the runbook comes out.