A Rollback Runbook for a Prompt That Passed Tests but Failed in Production

Every evaluation suite is a sample, and production is the population. The prompt change that passed 200 golden cases and then failed on real traffic is not a testing failure so much as an inevitability, which is why the response to it should be a runbook rather than an investigation.

Declaring it: what counts as a prompt incident

Prompt regressions do not announce themselves with a 500. The signals are indirect and each has a different lag, so define the triggers in advance and attach a number to each:

• Schema or parse failure rate on the prompt’s output. Fastest and most reliable signal, because it is a hard check. Any material rise after a prompt change is a rollback trigger with no further discussion.
• Downstream error rate. The service that consumes the output starts rejecting more of it. Often precedes the human signals by an hour.
• Refusal or empty-answer rate. A prompt edit that tightened a constraint can cause the model to decline work it should do. Count refusals as a first-class metric; they usually look like successes to everything else.
• Output length distribution shift. A change in mean output tokens is a cheap proxy for a change in behaviour, and it moves before any human notices.
• Human signals — thumbs-down rate, support volume, escalations. The most meaningful and the slowest, often by hours or days.

Write the threshold for each next to the metric, and make the decision rule explicit: any one hard signal, or any two soft signals, declares an incident. The purpose of a written threshold is to remove the argument about whether this is bad enough yet, which is where most of the delay in a real incident actually goes.

The first ninety seconds

1. Declare, and say what changed. One message with the symptom, the metric, and the suspected prompt version. Naming the version immediately is what stops three people independently investigating the model provider.
2. Check whether anything else changed. Look at the deploy timeline for the same window. A prompt change and a retrieval change landing together is common, and rolling back the wrong one costs you the whole first attempt. If two changes are in the window, roll back the prompt first anyway — it is the cheapest to reverse and the easiest to re-apply.
3. Capture failing examples before you touch anything. Thirty seconds of work, and it is the step that is always skipped. The next section says exactly what to capture.
4. Do not attempt a fix-forward. The instinct to “just add one line to the prompt” is strong and it is wrong, because you cannot evaluate the fix while the incident is running. Roll back, then fix at a normal pace with the failing cases in hand.

Rolling back, and confirming it took

1. Set the pointer or the kill switch to the previous version. One command. If your prompt is read at request time from a versioned store, this is the mechanism in rolling back a bad prompt deploy; if it is behind a flag, it is the kill switch in feature-flagging a prompt change. If it is neither, your rollback is a deploy and you should say so in the incident channel immediately, because the expected time is now twenty minutes rather than one.
2. Confirm from live traffic, not from a dashboard that says the write succeeded. Query the last minute of completion logs grouped by prompt version hash. Expect only the old hash. Two common surprises: one region or one deployment still shows the new hash because it reads a different store, or a cached value has a longer TTL than the one you tested.
3. Watch the fast signal recover. Schema failure rate should return to baseline within a couple of minutes. If it does not, the prompt was not the cause and the incident continues — re-open the “what else changed” question rather than rolling back further.
4. Account for the tail. In-flight requests, open streams and queued jobs are still on the old version. Get the counts; the in-flight middle state covers the policy. The incident is not over until that tail has drained, and saying so out loud stops somebody declaring all-clear while a backfill is still running.
5. Identify affected users while the data is fresh. Filter production records on the bad version hash for the affected window. That list is what support needs and it is much harder to reconstruct a week later.

Preserving the evidence you are about to destroy

Rolling back removes the only environment in which the bug reproduces. If you do not capture inputs first, the post-incident work becomes guesswork and the fix cannot be verified against anything real.

Capture, for a sample of failing requests: the request id, the rendered prompt or its hash, the full model output, the model and sampling parameters, the tool definitions if any were in play, and whatever downstream validation error was produced. Aim for something like fifty examples across the range of failures rather than five hundred of the same one — the variety is what tells you whether this is one bug or three.

# Pull failing examples into a candidate fixture set before rolling back.
# Redact before this leaves the incident channel: these are real user inputs.
psql -Atc "
  select request_id, prompt_rendered_hash, model, temperature,
         output_text, validation_error
  from llm_requests
  where prompt_template_hash = '9f2c81ad0b41'
    and validation_error is not null
    and created_at > now() - interval '2 hours'
  order by random() limit 50
" > incident-412-failures.tsv

Two cautions. These records contain user content, so the same handling rules apply as to any production data — redact before they move anywhere, and redact again before they become permanent test cases, since a fixture file outlives the incident by years. And sample randomly rather than taking the first fifty, or you will get fifty rows from one minute and one customer.

Closing the gap that let it through

The point of the captured examples is that they become the test that would have caught this. Work through them in order:

1. Reproduce offline. Run the captured inputs against the bad version and confirm the failure appears. If it does not, the prompt was not the whole cause and the real cause is still live — a retrieval change, a model update on the provider’s side, a truncation that only happens at certain context lengths.
2. Ask what class of input this was. Not “add these fifty cases” — that fixes fifty cases. The useful question is what property they share that your suite had none of: a language, a document length, an empty field, a user who pastes markdown tables, a locale with different date formats.
3. Add the class, not the instances. Extend the golden dataset with cases covering the property, drawn from production rather than written by hand. Hand-written cases inherit the same blind spot that let this through, because the same person wrote both. This is the general problem in evaluation blind spots.
4. Ask why the canary did not catch it. If the change went to 5% first and the regression was invisible, then either the traffic slice did not include the affected class — likely, if it was regional or tied to one large account — or the metric being watched was the slow human one. Both are fixable and both are more valuable than the prompt fix itself.
5. Record the rollback time. From first signal to confirmed recovery, plus the tail. If it exceeded your target, the action item is about the mechanism, not the prompt. That is the finding most worth having, because it applies to every future incident rather than to this one.