Building a Migration Runbook for a Provider Cutover

A migration runbook is not a plan. A plan says what you intend to do; a runbook says what the person on shift does next, what has to be true before they do it, and what makes them stop. This page builds one you can paste into a document and fill in.

The shape of the document

Every row in a useful runbook has four fields, and a row missing any of them will be argued about at the worst possible time.

• Action — one imperative sentence naming the exact change. “Set the routing weight for the checkout summariser to 5% new provider”, not “begin the ramp”.
• Entry condition — what must be true before the action is allowed. This is what stops a ramp continuing on schedule through a signal nobody looked at.
• Abort condition — the measurable condition that reverses this step. A metric, a threshold, a window. Not “quality degrades”. How to derive these numbers from your own baseline is the whole of defining a rollback trigger.
• Owner — a named person, not a team. The point of a name is that somebody has to be woken up, and “platform” cannot be woken up.

One more property that is easy to miss: every step must be reversible by a single change that does not require a deploy. If reverting the 25% step means shipping code, the runbook is a fiction, because a deploy during an incident takes longer than the incident is willing to wait. In practice that means the provider choice and the traffic split live in configuration or a feature flag, read at request time, with a documented default. Get that property before you write anything else down.

Phase 0: everything that happens before any traffic moves

Most of the runbook is phase 0, and teams that go badly are usually teams that started at phase 1. Work through these in order.

1. Freeze a baseline. Take one representative week of traffic and record, per route: request volume, p50 and p95 latency, error rate by class, mean output tokens, cost per thousand requests, and whatever quality metric you already have. This is the only defensible source for the thresholds you will write in the abort column. Do it before any code changes, so the numbers describe the system as it was.
2. Capture a replay corpus. A few hundred to a few thousand real requests, redacted, covering the shapes that matter — the long ones, the tool-calling ones, the non-English ones, the ones that historically failed. Sampling only the median request produces a migration that is validated on the traffic that was never going to break.
3. Build the adapter behind an interface you already have. One module, one function per call shape, provider-specific mapping entirely inside it. The general treatment is in provider-agnostic AI code. What matters for the runbook is that the interface is the rollback boundary.
4. Map the fields your observability depends on before you need them. Stop reasons, usage fields, request ids and error shapes all differ, and a dashboard that goes blank during the ramp removes your ability to read the abort conditions. Do this work now: migrating logging fields and cost dashboards after a migration.
5. Replay the corpus offline against both providers. Compare on the things that are cheap to check mechanically first — valid JSON rate, schema conformance, tool-call presence, output length distribution, truncation rate — before spending anything on human or model-graded quality. A large share of migration surprises are caught here, and truncation in particular is caught only if you look at the stop reason rather than at the text.
6. Write the abort conditions and get them agreed. Agreed means the person who will be on shift has read them and the person who owns the service has signed them off, in advance, in writing. A threshold agreed under pressure is not a threshold.
7. Rehearse the rollback. Flip the config to the new provider and back again in staging, with a stopwatch. You are measuring one number: how long from decision to traffic actually moving. That number goes at the top of the runbook, because it is the budget every abort condition is spending.

Phase 1-3: the ramp

The ramp exists because failure modes are not uniformly distributed across load. Rate limits, connection pool exhaustion, timeouts on the long tail and the cost of a verbose model at volume are all invisible at 1% and obvious at 50%. Each phase should sit long enough to cross at least one full traffic cycle — for most systems that is a business day, and for anything with a nightly or weekly batch it is that period plus one.

A shape that works: 1%, 5%, 25%, 50%, 100%, with the split applied per route rather than globally, and the least risky route first. Splitting per route matters because it makes the abort granular: a failing summariser gets reverted without dragging the four routes that are fine back with it.

Two decisions to make explicitly rather than by default. First, whether the split is sticky per user or per request. Sticky is better for any conversational surface, because a user whose thread alternates between two models mid-conversation experiences a style discontinuity that reads as a bug — and the same property makes A/B comparison honest, as discussed in variant stickiness. Second, whether you shadow. Shadow traffic sends the same request to both providers and compares, which is the only way to get a like-for-like quality signal on live inputs, and it doubles the spend for the shadowed fraction. Shadow a sample, not everything, and bound the window — the arithmetic is in the migration cost estimate.

The runbook table to copy

Fill in the bracketed values from your own baseline. The thresholds below are written as formulas against your recorded baseline rather than as numbers, because a number that did not come from your traffic is worse than no number.

PROVIDER CUTOVER RUNBOOK — <service> — <date>
Rollback latency measured in rehearsal: <T_rb> minutes
Config key: <flag name>   Default value: <old provider>

# | Action                          | Entry condition                       | Abort condition (any one)                      | Owner
--|---------------------------------|---------------------------------------|------------------------------------------------|-------
0 | Freeze baseline, capture corpus | none                                  | n/a                                            | <name>
1 | Offline replay, both providers  | corpus >= <N> reqs, covers <shapes>   | schema-valid rate < baseline - <x> pp          | <name>
2 | Deploy adapter, 0% traffic      | replay signed off                     | any error on health check path                 | <name>
3 | Route A to 1%, hold <period>    | dashboards show new-provider fields   | error rate > baseline + <x> pp over 15 min     | <name>
4 | Route A to 5%, hold <period>    | step 3 clean for full period          | p95 > baseline p95 * <k> over 15 min           | <name>
5 | Route A to 25%, hold <period>   | step 4 clean, cost/1k within <y>%     | cost per 1k requests > baseline * <m> over 1 h | <name>
6 | Route A to 50%, hold <period>   | step 5 clean, no 429s attributable    | 429 rate > <z>% of requests over 15 min        | <name>
7 | Route A to 100%, soak <S> days  | step 6 clean across one full cycle    | any of the above, or truncation rate > <t>%    | <name>
8 | Repeat 3-7 for routes B, C, ... | previous route at 100% for <S> days   | as above, per route                            | <name>
9 | Decommission old integration    | all routes soaked, notice period ok   | n/a — this step is irreversible                | <name>

ROLLBACK PROCEDURE (any abort condition, any step)
  1. Set <flag name> = <old provider>. No deploy.
  2. Confirm traffic moved: <dashboard/query>.
  3. Page <name>. Do not debug before step 1 is done.
  4. Record: step number, condition met, observed value, timestamp.
  5. Do not re-enter the ramp until the cause is understood and written up.

The one line in that table people delete is line 9’s note. Decommissioning is the only irreversible step, and it should not happen on the same day as the last ramp step, or in the same week if your notice period allows. Keep the old credentials valid and the old code path present until the soak has crossed a month-end, a peak, and whatever your system’s equivalent of a quarterly batch is.

Closing it out

Three things finish a migration, and skipping them is how the next one starts from zero again.

1. Compare the estimate to the outcome. Take the cost model you built at the start and fill in what actually happened: engineering days, eval sweeps, overlap spend, realised rate ratio. The delta is the most valuable artefact the migration produces, because it calibrates the next estimate.
2. Delete the dual path deliberately. A rollback flag that has been at one value for six months with no test covering the other value is not a rollback path, it is dead code that will not work when tried. Either keep it exercised — a periodic run of the replay corpus through the old provider — or remove it and say so.
3. Re-baseline. Every threshold in the abort column was derived from the old provider’s numbers. After the soak, record the new baseline and update the alerts that outlive the migration, or you will spend the next quarter alerting against a system that no longer exists.