Skip to content
Multigrid

Rolling out a prompt change

Canaries, shadow traffic, feature flags and rollback — treating a prompt edit as a deploy, because it is one.

A prompt edit changes the behaviour of production for every user, takes effect the moment it lands, and is usually reviewed by fewer people than a one-line config change. The gap between how consequential it is and how casually it ships is where most prompt incidents live. Nothing about the edit being English rather than TypeScript makes it safer to roll out unstaged; if anything it is worse, because the failure is rarely an exception and so nothing throws.

These pages treat the edit as what it is — a deploy — and work through the machinery that implies. How large a first slice actually buys you anything, what to watch besides the eval score, when to let a rollback fire itself and when that will cost you a false alarm every ninety minutes, how to compare two versions of a non-deterministic system without a human reading every output, and how to load-test the thing before it meets real traffic. The arithmetic is shown wherever a number is used, because most of the defaults people repeat here do not survive being derived.

Choosing a Canary Percentage for a Prompt Change

Why the choice between 1%, 5% and 20% is decided by how fast you can detect a total failure, not by statistical power.

10 min read

What to Compare Between Canary and Baseline Beyond the Eval Score

The secondary signals a prompt canary should watch — token counts, finish reasons, tool-call shape, refusals — and why the eval score is structurally blind to all of them.

10 min read

Rolling a Canary Back Automatically on an Error Rate Spike

A two-tier automatic rollback trigger, with the arithmetic showing why a naive error-rate threshold on a small canary fires by chance every ninety minutes.

11 min read

Running a Shadow Deployment Without Doubling Your Bill

How sampling and stratifying shadow traffic gets a confident read at a small fraction of the cost of mirroring everything, and where the hidden multipliers are.

10 min read

Comparing Shadow Traffic Outputs Without a Human Reading Every One

The cheap automatic checks that separate two prompt versions' outputs, and the base-rate arithmetic that decides whether a prefilter is worth running.

10 min read

Sequential Testing for a Prompt Change Instead of a Fixed-Horizon A/B Test

Why watching a fixed-horizon experiment destroys its error guarantee, and the three families of test that let you stop as soon as the evidence is in.

10 min read

Interaction Effects When Two Prompt Experiments Run at Once

Why overlapping prompt experiments leave the main effects unbiased but the interaction underpowered by exactly four times, and the two mistakes that do bias them.

10 min read

Sampling Rate for Human Review of LLM Output in Production

The arithmetic connecting a target margin of error to a review sample size, and why it shows that a percentage-of-traffic review policy is the wrong shape.

10 min read

Routing Only Low-Confidence Outputs to a Human Reviewer

Which confidence signals an LLM actually gives you, where each one is calibrated, and how to turn a score into a triage rule that fits a fixed reviewer budget.

10 min read

Building a Review Queue for Failed Prompt Test Cases

A minimal tool that turns a wall of failing regression output into a deduplicated, stably-identified queue with a three-way disposition.

10 min read

Scrubbing PII From Recorded LLM Test Fixtures Before Committing Them

A record-time scrubber plus a pre-commit backstop for VCR-style cassettes, covering the two things generic redaction misses: JSON message bodies and SSE frames.

10 min read

Load Testing an LLM API With k6

A k6 script for a chat completions endpoint, using an open workload model so that rising latency does not silently reduce the load you are applying.

11 min read

Load Testing a Streaming Endpoint With Locust

Why Locust reports a streaming request as fast and zero-length by default, and the script that measures time to first token, total time and inter-token gaps instead.

11 min read

Simulating a Realistic Prompt Length Distribution in a Load Test

Drive a load test from a sampled prompt-length distribution instead of one fixed prompt, so the latency and cost numbers it produces describe traffic you will actually serve.

9 min read

Load Testing Concurrent Tool-Calling Requests

How to load-test an agent loop, where one user request becomes several model calls plus your own tool executions, and the concurrency you configure is not the concurrency you get.

10 min read

Injecting a Fake Provider Timeout to Test Your Failover Path

A fault-injection test that makes a provider hang rather than error, and asserts that failover triggers on the timeout instead of leaving the request stuck.

9 min read

Testing Failover When Two Providers Are Down at Once

How to test the branch every failover suite skips: what your service does after the last configured fallback has also failed.

9 min read

Chaos Testing a Rate-Limit Response From a Provider

Inject a synthetic 429 and assert that your code backs off and retries with jitter rather than treating a rate limit as a hard failure or hammering straight through it.

9 min read

A Code Review Checklist for a Prompt Change Pull Request

What to look for when reviewing a pull request that edits a prompt, and what evidence the author should have attached before you are asked to approve it.

9 min read

Storing Prompts as Files Instead of Strings in Code, for Better Diffs

Why moving prompt text out of source literals and into files makes the wording change visible in review, and what a workable file layout and loader look like.

9 min read

Tracking Which Prompt Version Produced a Given Production Response

A logging convention that records a content hash of the rendered prompt on every response, so a support ticket can be traced back to the exact text that produced it.

9 min read

Rolling Back a Bad Prompt Deploy in Under a Minute

Move the prompt out of the deploy artefact and into a versioned store read at request time, so reverting is a config write rather than a build and a rollout.

9 min read

Feature-Flagging a Prompt Change So Rollback Is Instant

Ship both prompt versions in one deploy and let a flag decide which one serves each request, so turning the change off is an evaluation change rather than a release.

9 min read

What to Do About In-Flight Requests When You Roll Back a Prompt

The awkward middle state after a rollback fires: requests already running on the old prompt, streams half-delivered, and multi-turn conversations that started on the other version.

9 min read

A Rollback Runbook for a Prompt That Passed Tests but Failed in Production

A step-by-step incident runbook for the gap every evaluation suite has: production inputs the golden dataset never contained.

10 min read

A Compatibility Test Suite for a Self-Hosted vLLM Endpoint

A suite that pins down which parts of the OpenAI API your self-hosted vLLM server actually implements, so an upgrade that changes the surface fails a test rather than a customer request.

10 min read

Other topics