Defining a Rollback Trigger for a Provider Migration

“We will roll back if quality drops” is not a trigger. It has no metric, no threshold, no window and no owner, so at the moment it is needed it becomes a discussion — which is exactly the thing a trigger exists to prevent.

The four parts of a usable trigger

A rollback trigger is a condition a monitor could evaluate without human judgement. Write each one in this form and the missing pieces become obvious:

IF   <metric>  <comparison>  <threshold>
FOR  <window>
ON   <scope>
THEN <action>, owned by <person>

A worked instance, with every field filled from a baseline rather than from a preference:

IF   5xx-equivalent error rate on route /summarise
     > baseline_error_rate + 0.5 percentage points
FOR  15 consecutive minutes
ON   the new-provider share of traffic only
THEN set provider flag to old, page the on-call owner

Two details in that example do real work. The comparison is against a recorded baseline rather than against zero, because the old provider also had an error rate and a trigger that ignores it will fire on normal background failure. And the scope is the new provider’s share only: during a ramp your global error rate is a blend, so a global threshold is diluted by exactly the fraction of traffic you have not moved yet. At 5% traffic, a new provider failing every single request moves a global error rate by five points — which many teams would not alert on.

Which metrics make good triggers

A good trigger metric is available within minutes, attributable to the provider, and low-noise at your volume. That rules out most quality metrics as primary triggers and promotes some unglamorous ones.

Strong candidates

• Error rate by class, split by provider. Separate transport failures, rate limiting, and model-side refusals or content filtering — they have different causes and different fixes, and blending them hides which one is happening.
• p95 latency, not mean. A mean is dragged by a handful of slow requests and hides a bimodal distribution. For streaming surfaces, time to first token deserves its own trigger, because a regression there is what a user feels even when total time is unchanged.
• Truncation rate. The share of responses whose stop reason indicates the token limit was reached rather than a natural end. This is a first-class migration signal because output length differs between models and a limit that was generous for one is tight for another. It is also nearly free to compute — the field is already in the response.
• Schema conformance. For any structured-output path, the share of responses that parse and validate. This catches the single most common migration break and it is binary, so it is not noisy.
• Cost per thousand requests. A blunt but honest proxy for verbosity and retry storms combined. Evaluated over an hour rather than minutes, since billing signals are lumpy.

Weak candidates, and why

• Model-graded quality scores. Too slow, too expensive per sample, and noisy enough that a threshold tight enough to catch a real regression also fires on nothing. Use them in the offline replay before the ramp, and as a slower daily check during it, not as a minute-scale trigger.
• User-reported complaints. A real signal and a terrible trigger, because the latency between a bad response and a ticket is hours to days. Track it; do not gate on it.
• Anything with no baseline. If you did not record the metric before the migration, you cannot set a defensible threshold on it during one. That is an argument for the baseline freeze in the migration runbook, not for guessing.

Deriving the threshold from your baseline

The question “what number should the threshold be” has no general answer, but it has a general method. Take the metric over your baseline week, at the granularity you will evaluate it — if the trigger is a 15-minute window, compute the metric per 15-minute bucket across the whole week. You now have a distribution. Set the threshold above the worst bucket that was not an incident.

METHOD (per metric)
  1. bucket the baseline week at the trigger's window size
  2. drop buckets covering known incidents
  3. take the maximum of what remains          -> B_max
  4. threshold = B_max + margin

  margin is a judgement call. A useful default is the gap
  between the 99th percentile bucket and B_max: if normal
  operation already spans that much, the trigger must clear it.

This procedure has one property worth stating plainly: it produces a threshold that would not have fired during a normal week before the migration. That is the entire point. A trigger that would have fired on the old system is a trigger that will fire on the new one for reasons unrelated to the new one, and after the second false rollback nobody will act on the third.

For rate metrics, check the threshold against your volume before you accept it. If a route serves 40 requests in a 15-minute bucket, a threshold of “error rate above 2%” is a threshold of “one request failed”, which will happen constantly. Either widen the window until the bucket holds enough requests for the rate to mean something, or switch to an absolute count. The rule of thumb is that the threshold should correspond to at least five events, not one.

The window is where triggers go wrong

The window has to satisfy two opposing constraints, and naming them separately makes the trade-off tractable.

• Long enough that noise does not cross it. A single-sample trigger on a p95 latency will fire on one slow request and on every provider hiccup that recovers in seconds. Requiring the condition to hold for several consecutive evaluation periods is the standard fix and it costs nothing but the delay.
• Short enough that the damage is bounded. The real budget is the window plus the rollback latency you measured in rehearsal plus the time to notice. A 30-minute window on a system where rollback itself takes 10 minutes means 40 minutes of degraded traffic minimum, and that has to be acceptable to whoever owns the user experience.

The practical resolution is tiered triggers rather than one number. A fast, high-threshold trigger catches catastrophe: if the error rate on the new provider exceeds some large value for two minutes, roll back immediately, because nothing that severe is noise. A slower, tighter-threshold trigger catches degradation: if p95 exceeds the baseline multiple for thirty minutes, roll back. And a daily check catches drift that no minute-scale metric sees at all — cost per request, truncation rate, quality on the replay corpus.

Add one more that is not a metric: a calendar trigger. If the ramp has not reached the next phase by a stated date, that is itself a condition worth acting on, because a migration that stalls at 25% while a notice period runs down is heading for a cutover with no rollback path. Deciding to pause and re-plan is a legitimate outcome; drifting into a forced cutover is not.

Who pulls it, and what happens after

Two rules make the trigger real rather than decorative, and both are about authority rather than measurement.

The first: the person on shift executes the rollback without seeking approval. If pulling the trigger requires finding a decision-maker, the trigger is advisory and the migration’s worst-case duration is however long it takes to reach that person at three in the morning. Pre-agreement is what buys this — the conditions were signed off in advance precisely so that meeting them is not a judgement call.

The second: rolling back is not a failure and must not be treated as one. The behaviour to avoid is the engineer who suspects the trigger has been met and spends twenty minutes proving it to themselves first, because they know a rollback will be questioned. Roll back first, diagnose after — the data you need to diagnose is in the logs either way, and the traffic is not.

Afterwards, write down four things while they are fresh: which condition was met, the observed value against the threshold, what the logs showed at the boundary, and whether the threshold was right. That last one is the interesting one. A trigger that fired on something harmless needs its threshold or window revised before the next attempt; a trigger that did not fire on something harmful needs a new metric. Both are cheap to fix between ramp attempts and impossible to fix during one.