Canary and Blue-Green Deploys for Model Changes

A code deploy either works or throws. A model deploy usually does neither: the new version returns 200s at the same latency and is slightly worse at the thing you care about. That is why model rollouts need a quality gate as well as a health gate, and why the honest question about a canary is not “did it error?” but “have I seen enough requests to tell?”

Why model deploys are not code deploys

Three properties break the usual assumptions.

• Failure is silent and distributional. A 2% drop in extraction accuracy produces no errors, no latency change and no alert. It shows up as support tickets three weeks later.
• Output is non-deterministic. You cannot diff two responses and conclude anything from one pair. Comparison has to be statistical, over a sample.
• Capacity is the constraint on the strategy. Running blue and green simultaneously means holding two full sets of GPUs. For a large model that doubles the most expensive line in the budget for the duration of the rollout, which is a real reason to prefer a small canary over blue-green.

Add one more, which applies when you are switching between hosted models rather than your own weights: providers update models under a stable name. Silent model updates covers that case, and the defence is the same eval gate applied on a schedule rather than on a deploy.

Blue-green, canary, and shadow

They compose, and the sensible default is to compose them: shadow first to catch crashes and gross regressions on real inputs, then canary to catch what shadowing cannot see, then promote. Shadow traffic for LLMs covers the duplication mechanics, including the trap of shadowing requests that have side effects.

Splitting traffic deterministically

The split must be sticky per user, not per request. A user whose conversation alternates between two model versions gets an inconsistent assistant, and any metric you compute over the session is contaminated. Hash a stable identifier into a bucket:

import hashlib

def variant(user_id: str, canary_percent: int, salt: str = "chat-model-2026-08") -> str:
    """Stable assignment: the same user always lands in the same bucket
    for a given salt. Change the salt to re-randomise a later experiment."""
    h = hashlib.sha256(f"{salt}:{user_id}".encode()).digest()
    bucket = int.from_bytes(h[:4], "big") % 100        # 0..99
    return "canary" if bucket < canary_percent else "stable"

Two details. The salt means a second rollout does not reuse the same unlucky users, which otherwise correlates your experiments. And sha256 rather than Python’s built-in hash, because the built-in is randomised per process and would reassign every user on restart.

At the infrastructure layer the same split can be expressed by two Deployments behind one Service with proportional replica counts, or by a weighted route in a service mesh or ingress. The replica-count method is the crudest — a 10% canary means one replica in ten, so you cannot express 1% without ten replicas — but it needs nothing installed. Whichever layer does the splitting, record the variant on the request so that every log line, trace and evaluation can be grouped by it. Without that label the canary produces no evidence at all.

The eval gate

The promotion criterion is a set of thresholds decided before the rollout, checked automatically, with no judgement call at 2am. Three tiers, evaluated in order:

1. Offline eval, before any traffic. The frozen eval set from your eval set run against the candidate. Absolute gate: below threshold, the rollout does not start. This is the only tier that costs nothing to fail.
2. Operational metrics, from the first canary requests. Error rate, p95 latency, time to first token, tokens per second, truncation rate, refusal rate, cost per request. These move fast and catch the gross failures within minutes.
3. Quality metrics, over the canary window. Whatever proxy you have for correctness on live traffic: schema-validation pass rate, tool-call success, retrieval-grounded citation checks, thumbs-down rate, human review of a sample. These are slow and they are the ones that decide whether the model is actually better.

Automate tiers one and two completely. Tier three is where the sample size question bites, and where most teams promote on a number that could not possibly have been significant.

How long the canary must run

Suppose your quality proxy is a pass rate — schema valid, tool call succeeded, judge said acceptable. You want to detect a regression of size Δ in that rate. The standard two-proportion sample size, with all assumptions labelled:

n per variant ≈ 2 × p(1−p) × (z_α/2 + z_β)² / Δ²

  p    baseline pass rate
  Δ    the smallest regression you want to detect, in absolute terms
  z    normal quantiles; for 95% confidence and 80% power,
       z_α/2 = 1.96 and z_β = 0.84, so (1.96 + 0.84)² = 7.85

Worked, with p = 0.90:

  Δ = 0.05 (detect a drop from 90% to 85%):
      n = 2 × 0.09 × 7.85 / 0.0025 = 565 requests per variant

  Δ = 0.02 (detect 90% → 88%):
      n = 2 × 0.09 × 7.85 / 0.0004 = 3,533 per variant

  Δ = 0.01 (detect 90% → 89%):
      n = 2 × 0.09 × 7.85 / 0.0001 = 14,130 per variant

Note the 1/Δ² : halving the effect you want to catch quadruples the sample.

Now convert that into a canary duration, which is the number you actually need:

  total traffic ......... 20 requests/second = 1.73 M/day
  canary share .......... 5%              = 86,400 canary requests/day
  requests with a usable quality label ... 30%  (not every request has one)
                                          = 25,920 labelled/day

  To detect Δ = 0.02, needing 3,533 labelled canary requests:
      3,533 / 25,920 ≈ 0.14 days ≈ 3.3 hours

  At a 1% canary instead:
      canary labelled/day = 5,184 → 3,533 / 5,184 ≈ 16 hours

This is the arithmetic that turns “run the canary for a while” into a decision. If the answer is longer than you are willing to wait, your options are honest ones: raise the canary percentage, accept a larger detectable Δ, label more requests, or lean harder on offline eval where you control the sample. Statistics for LLM evals goes further into the multiple-comparison problem you create by watching six metrics at once.

Rollback criteria, written in advance

Write the abort conditions into the rollout plan before it starts, in the form “if X for Y minutes, revert”. The point of writing them down is that at 2am, watching a number wobble, nobody has to decide what counts as bad.

# rollout.yaml — the plan, reviewed with the change
steps:  [1, 5, 25, 50, 100]     # percent of traffic
dwell:  [1h, 4h, 12h, 12h, -]   # minimum time at each step

abort_if:
  - metric: error_rate_5xx      op: ">"  value: 0.5%   for: 5m
  - metric: p95_latency_ms      op: ">"  value: 1.25   relative_to: stable  for: 10m
  - metric: schema_valid_rate   op: "<"  value: 0.98   relative_to: stable  for: 30m
  - metric: cost_per_request    op: ">"  value: 1.15   relative_to: stable  for: 1h
  - metric: thumbs_down_rate    op: ">"  value: 1.30   relative_to: stable  for: 4h

on_abort:
  - set canary traffic to 0        # seconds; the stable pods never went away
  - keep canary pods running       # for diagnosis, not for traffic
  - page the on-call, do not auto-promote again without a human

The last two lines are the ones people leave out. Keeping the canary pods running after an abort preserves the evidence — logs, traces, a live process to inspect — and setting traffic to zero rather than deleting the deployment makes the rollback take seconds rather than a scheduling cycle. What to do next is in the on-call runbook, which has a page for exactly this symptom.