Migrating an On-Call Escalation Policy for AI Incident Types

The runbook gets rewritten during a migration. The escalation policy usually does not, because it lives in a different tool and belongs to a different person. The result is an on-call engineer classifying a brand-new failure mode against a severity table written for the previous provider, at three in the morning.

Why the policy lags the provider

An escalation policy is a mapping from observable signals to a severity and a paging decision. Both halves are provider-shaped. The signals are fields in a response body and headers on an error; the thresholds were set from a baseline measured on the old provider. Change the provider and some signals stop existing, some appear, and every threshold is calibrated against traffic that no longer flows.

The specific harm is misclassification in the safe-looking direction. A failure with no matching row gets classified as whatever it most resembles — usually “degraded, monitor” — and quietly runs for hours. The policy did not fail loudly; it just had no opinion.

Failure modes a migration introduces

These are the classes that reliably have no row in a policy written for a single previous provider:

• Schema validation failure rate. Structured-output enforcement differs between providers. Outputs that always validated may now fail intermittently, and if your client retries silently the only symptom is latency and cost.
• Stop-reason distribution shift. The field naming why generation ended differs by provider and so does its value set. A rise in length-limited terminations means silent truncation; a rise in refusal-shaped terminations means a content-policy boundary moved. Both are invisible unless you record the field.
• Cache hit-rate collapse. Not an availability incident. It is a cost incident that looks like nothing until the invoice, and it is the most common consequence of a cutover — see what a migration does to cache TTL assumptions.
• Tokenizer-driven truncation. The same text is a different number of tokens on a different model. An output budget that fitted comfortably can start clipping the tail of long responses, which surfaces as malformed JSON rather than as a length error.
• Rate-limit shape change. Whether a limit is per request, per input token, per output token, or several at once differs, and so does whether a retry-after header is present. A backoff policy that assumed the header exists degenerates into a fixed sleep.
• Streaming disconnect rate. Long generations dropped mid-stream may be handled entirely differently by the target’s infrastructure and by any proxy in front of it.

The severity matrix

A policy row needs four columns and no prose: the signal, the expression, the severity, and who wakes up. Thresholds below are written as symbols deliberately — substitute values measured from your own baseline, because a threshold copied from someone else’s policy is a threshold that fires at the wrong time.

| signal                  | expression                              | sev  | page |
|-------------------------|-----------------------------------------|------|------|
| provider 5xx rate       | rate5m > B_5xx * 3                       | SEV1 | yes  |
| all providers erroring  | healthy_provider_count == 0              | SEV1 | yes  |
| schema validation fails | rate15m > B_schema * 5                    | SEV2 | yes  |
| truncated outputs       | stop_reason == length, rate15m > B_trunc  | SEV2 | yes  |
| refusal-shaped stops    | rate1h > B_refuse * 4                     | SEV2 | yes  |
| p99 latency             | p99_15m > B_p99 * 2                       | SEV2 | yes  |
| 429 rate                | rate15m > B_429 * 3                       | SEV3 | no   |
| cache hit rate          | hit_rate_1h < B_cache * 0.5               | SEV3 | no   |
| stream disconnects      | rate1h > B_stream * 3                     | SEV3 | no   |
| cost per request        | cost_24h > B_cost * 1.5                   | SEV3 | no   |

B_* are baselines measured over the two weeks before cutover, on the
provider that is being migrated FROM, and re-measured on the target
during dual-running. Do not carry a threshold across the cutover
without re-measuring it.

Two rows are worth defending explicitly. Cache hit rate and cost per request are SEV3 and do not page, but they must be rows — without them, the failure mode that actually costs money during a migration has no owner. And truncation is SEV2 rather than SEV3 because its downstream symptom is corrupt data, not a slow page.

Two severities that exist only while dual-running

While both providers serve traffic, a failure has an extra dimension: which side it happened on. Most policies have no vocabulary for this and consequently either page for every shadow-traffic error or ignore all of them.

• New provider only. The target fails and the incumbent is healthy on the same input. Users are unaffected because you have not cut over. This is SEV3, routed to the migration owner during business hours, and it is the single most valuable signal you will collect — it is a free discovery of a target-specific edge case. Feed it straight into the edge-case suite.
• Both providers. Both fail on the same input. This is not a provider incident at all; it is a bug in your prompt, your schema, or the input itself. Classify it by user impact under the normal matrix and do not let anyone spend the night blaming a vendor.

Delete both rows on the day dual-running ends. A policy carrying dead rows teaches on-call to skim.

Rewriting the policy

1. Inventory the current AI rows. List every alert that fires on a model call. For each, write down the exact field or header it reads. Most policies have three or four rows and a great deal of unwritten convention.
2. Check each signal exists on the target. Send one request per failure class against the target and record the actual response and headers. A row reading a field the target does not emit is a row that will never fire, and it will look healthy on the dashboard forever.
3. Add the new-mode rows. From the list above, keep the ones the target can actually produce. Write the expression, leave the threshold as a symbol.
4. Measure the baselines. Run for two weeks and take a high percentile of each signal as the baseline. Thresholds set before you have a baseline are guesses that will either page constantly or never.
5. Add the two dual-running rows with an end date in the row itself, and a calendar reminder to remove them.
6. Write the rollback trigger as an expression. “Roll back if quality degrades” is not a trigger, because at 3am nobody will own that decision. Something like schema_fail_rate_15m > 2% OR p99 > 2x baseline for 30m is, and the runbook step next to it should be a single command.
7. Rehearse once. Force one failure of each new class in staging — a deliberately unsatisfiable schema, an input long enough to truncate, a revoked key — and confirm the right row fires and the right person is paged. A policy that has never fired is a document, not a control.