Canary Releases for Model Migrations

A model migration is a production change with an unusually wide blast radius and an unusually vague failure signature. The canary is the same instrument you would use for any risky deploy — the difference is in what you watch, and in how long each stage has to run before the watching means anything.

Sticky assignment, not per-request coin flips

The obvious implementation — roll a random number per request — is wrong for conversational features. A user whose turns alternate between two models gets a conversation with two personalities, two formatting styles and two levels of caution. Worse, it makes the comparison meaningless: your two groups are not two populations, they are the same population sampled twice.

// Assignment is a pure function of a stable key. No state, no coordination,
// and every service in the request path computes the same answer.
export function canaryModel(cfg: CanaryConfig, key: string): string {
  const bucket = fnv1a32(cfg.experimentId + ":" + key) % 10000;
  return bucket < cfg.rolloutBps ? cfg.candidate : cfg.baseline;
}

// Choose the key by what must stay consistent:
//   conversation.id  — a single conversation never switches mid-thread
//   tenant.id        — a whole workspace sees one model (best for B2B)
//   user.id          — an individual is consistent across conversations

Salting the hash with an experimentId matters more than it looks. Without it, the same tenants land in the canary bucket for every experiment forever — so your early adopters are permanently your guinea pigs, and any correlated problem they have contaminates every rollout you run.

Choosing the key is a product decision more than a technical one. For a B2B product, hash on the tenant: a workspace where two colleagues get visibly different answers to the same question generates support tickets that have nothing to do with quality. For a consumer product with independent sessions, the conversation id gives you finer-grained traffic and therefore faster results. What you must not do is switch keys midway through a rollout, because that reshuffles everyone and discards the comparison you had been accumulating.

One consequence of sticky assignment worth planning for: your two arms are not statistically identical populations. Whichever tenants landed in the canary bucket have their own request mix, and at small percentages that difference can be larger than the effect you are looking for. The guard is to compare each arm against its own recent history as well as against the other arm — a candidate that looks worse than baseline but no worse than the same tenants looked last week is telling you about the tenants, not the model.

The ladder, and how long each rung takes

Stage sizes are usually picked as round numbers. It is better to pick them from what you need to observe, using the same sample-size arithmetic as regression detection: to detect an absolute change δ in a rate p, you need roughly 7.84 · 2p(1−p) / δ² requests per arm.

The honest consequence of this arithmetic: for a low-traffic feature, a canary cannot detect a subtle quality change at any stage size, and pretending otherwise wastes a week. For those, lean on shadow traffic and offline evals, and use the canary only as a smoke test.

Rollback triggers, specified

A trigger that says “if quality drops” is not a trigger. Each one needs a metric, a comparison, a threshold, a minimum sample and an action.

triggers:
  - name: hard_errors
    metric: error_rate                 # 5xx, timeouts, refused connections
    compare: candidate_vs_baseline
    threshold: absolute +0.5pp
    min_samples: 200
    window: 10m
    action: rollback                   # automatic, no human

  - name: schema_validity
    metric: schema_failure_rate
    compare: candidate_vs_baseline
    threshold: relative +25%
    min_samples: 1000
    window: 1h
    action: rollback

  - name: latency_p95
    metric: ttft_p95_ms                # TTFT, not total: streaming
    compare: candidate_vs_baseline
    threshold: relative +30%
    min_samples: 500
    window: 30m
    action: hold                       # stop advancing, page nobody

  - name: unit_cost
    metric: cost_per_request
    compare: candidate_vs_baseline
    threshold: relative +20%
    min_samples: 1000
    window: 1h
    action: hold

  - name: refusal_rate
    metric: refusal_rate
    compare: candidate_vs_baseline
    threshold: absolute +2pp
    min_samples: 2000
    window: 2h
    action: rollback

Two design choices in there are worth stating explicitly. Comparisons are always candidate-versus-baseline over the same window, never candidate-versus-a-fixed-number — otherwise a provider-wide slowdown rolls back a perfectly good canary. And not every trigger rolls back; hold is the right action for cost and latency, which are decisions rather than emergencies.

The absence of a quality trigger is deliberate. Rubric-judged quality is sampled, lagging and noisy — it cannot support an automatic action on the timescales above, and wiring it to one produces rollbacks caused by resampling. Quality belongs in the promotion decision between stages, made by a person looking at the graded sample, not in the automation that runs between them.

Note also that latency_p95 keys on time to first token rather than total duration. A candidate that writes longer answers has a worse total duration by construction, and rolling back for that would reject models on verbosity while calling it a latency regression. If verbosity is the problem, it shows up in unit_cost, which is where you actually want to have that argument.

Not rolling back on noise

The min_samples field is what stops a canary from oscillating. Without it, the first ten requests at 1% will occasionally contain two errors, the trigger fires at a 20% error rate, and the rollout is abandoned for a reason that was never real.

Two more guards are worth having. Require the condition to hold over two consecutive evaluation windows before an automatic rollback, so a single provider blip does not undo a week’s work. And rate-limit the automation itself: if a canary has rolled back twice, stop and escalate rather than trying a third time, because a system that automatically retries a failing migration will eventually do it at 03:00 unattended.

Rollback has to be a config change

If rolling back means reverting a commit and waiting for CI, your rollback time is your deploy time, and the whole exercise is theatre. The candidate model and the rollout percentage belong in a flag that resolves at runtime, readable from a local cache, with an embedded default so a configuration outage does not become an inference outage.

Test it before you need it: set rolloutBps to 0 in staging and time how long until the last request routed to the candidate. If that number is more than a minute, fix it now rather than discovering it during the migration.

Rollback also has to be more than routing. The candidate may have needed a different max_tokens, a reworded tool description or a slightly different prompt to behave well — and if those shipped as code while the model shipped as config, reverting the model leaves you running the baseline with the candidate’s scaffolding. Bundle them: the flag should select a named configuration, not a model string.

Finally, write down what the migration was for before you start. A canary tells you whether the candidate is worse; it does not tell you whether it is better in the way you cared about. If the goal was cost, the promotion criterion is a cost number with quality held flat; if it was quality, it is the reverse. Deciding that afterwards, while looking at the results, is how a migration gets promoted on the one metric that happened to move.