Why a Prompt That Worked Stops Working
9 min read · updated August 4, 2026
A prompt is a static string. It cannot rot. When one that worked for months stops working, exactly one of four things changed — the model, the input distribution, the surrounding context, or the definition of correct — and each has a different fix. Guessing wrong costs a week.
Prompts do not decay; four things change
“Prompt decay” is a useful name for a real phenomenon and a misleading description of it. Nothing about the prompt changes. What changes is one of the four inputs the prompt was implicitly tuned against, and because the tuning was implicit, nobody wrote down what it depended on.
That is the whole diagnostic problem: the prompt was fitted to a distribution that was never recorded, so the first task is to work out which part of that distribution moved.
The four mechanisms
| Mechanism | Description |
|---|---|
| 1 · The model changed | The endpoint you call is not serving the weights it served in March. A version alias moved, a provider swapped a quantisation, or a deprecated model was silently redirected to its successor. The prompt is unchanged, the function behind it is not. |
| 2 · The input distribution moved | Users are sending different things. A new customer segment, a new integration, a new language, longer documents. The prompt still works on the inputs it was tested against and those are now a smaller share of traffic. |
| 3 · The surrounding context changed | The prompt is one part of a request that also contains retrieved documents, conversation history and tool schemas. Any of those growing, re-ordering or changing format alters what the model sees, even though the instruction text is byte-identical. |
| 4 · The definition of correct changed | Nothing technical moved. Expectations did — a stakeholder now wants shorter answers, a new compliance rule forbids a phrasing, a downstream consumer tightened its schema. This is the most common and the least often recognised. |
Telling which one you have
Run these four checks in order. Each is cheap, and each eliminates one mechanism outright.
- Replay a fixed set of old inputs against the current endpoint. If archived inputs that used to pass now fail, the model or the context changed and the input distribution did not. This single check separates mechanisms 1 and 3 from 2 and 4, and it is why keeping fifty archived request bodies is worth more than any monitoring dashboard.
- Pin the model to an explicit dated version and replay again. If the failures disappear, it is mechanism 1. If your provider does not expose a pinned version, that is itself the finding — see silent model updates.
- Diff the assembled request, not the prompt. Log the full message array for a failing request and a passing archived one, and diff them. Growth in retrieved context, a reordered system message or a new tool schema shows up immediately. This is mechanism 3, and it is the one people miss because they are reading the prompt template rather than the rendered request.
- Compare the failing inputs against the archived ones. Length distribution, language, and structure. A visible shift is mechanism 2. No shift and no technical change means mechanism 4, and the fix is a conversation rather than a commit.
The order matters because steps 1 and 2 are automatable and steps 3 and 4 are not. Do not start by rewriting the prompt: a rewrite that happens to help masks the cause and buys a repeat of the same incident.
Detecting it before a user does
All four mechanisms are detectable from signals you can emit on every request. None of them requires a human reading outputs.
| Signal | Description |
|---|---|
| Parse or schema-validation rate | The fastest-moving indicator there is. A structured-output prompt that starts failing to validate will show here days before anyone files a ticket, and it is one boolean per request. |
| Output length distribution | Median and p95 completion tokens per prompt template. A model swap almost always moves this, and it moves before quality complaints do. |
| Refusal and empty-completion rate | Per template. Catches both a model change and an input-distribution change towards material the model declines. |
| Assembled context size | Prompt tokens per template, tracked over time. Mechanism 3 is invisible in every other signal and obvious in this one. |
| A pinned regression set, run on a schedule | Twenty to fifty archived inputs with known-good outputs, run nightly against the live endpoint. This is the only signal that detects a quality change with no structural symptom. |
The first four are aggregate and free. The fifth costs a few cents a night and is the only one that catches mechanism 1 when the model change is subtle. Build it the same way you would any other regression suite.
The guards, one per mechanism
- Against a changed model: pin an explicit version in configuration, never an alias, and treat a version bump as a code change that runs the regression set. Where a provider offers no pinning, record the model string the response reports and alert when it changes.
- Against a moved input distribution: sample real inputs into the regression set continuously, so the set ages with traffic instead of freezing at launch. A set that never changes is measuring last year’s product.
- Against changed surrounding context: assert a budget. Fail the request — loudly, in development — when the assembled prompt exceeds the size the template was tuned for. Budgeting context explains how to divide it.
- Against a changed definition of correct: version the prompt and the acceptance criteria together, in the same commit. If the criteria live only in somebody’s head, every future disagreement is unresolvable. Prompt versioning covers the mechanics.
Fixing it without starting the cycle again
Once the mechanism is identified, there is a strong temptation to fix it by adding sentences to the prompt. That is how the next incident is created, because each added sentence is another implicit dependency nobody recorded.
- Reproduce on the archived inputs first. A fix validated only against the newly failing inputs will over-fit to them, and the regression it causes on the old ones will surface as a second incident a fortnight later.
- Prefer a structural fix to a textual one. A schema that forbids a preamble is more durable than an instruction not to write one. A validator with a repair pass survives a model change; a carefully worded sentence often does not.
- Change one thing. Prompt text, model version and retrieval configuration are three variables. Changing two at once produces a result you cannot attribute, and attribution is the entire point of having done the diagnosis.
- Record what the change assumes. In the same commit as the prompt: which model version it was tuned against, what the assembled prompt size was, and what “correct” meant on that day. This is the record whose absence caused the diagnosis to be hard, and writing it is the only part of this page that prevents a recurrence rather than detecting one.
- Add the failing inputs to the regression set before closing. Not after. The set that grows from real failures is the one that catches the next occurrence; the set that was written at launch will not.
Mechanism 4 — a changed definition of correct — deserves one extra step: write the new criteria down and get them agreed before touching anything. A prompt tuned against an unstated expectation is the starting condition for the whole cycle, and it is where most of these incidents began.
What none of this catches
Three honest limits, because a detection scheme that overstates its coverage is worse than none.
- Slow quality drift within valid output. If answers get gradually blander while remaining well-formed, correct-looking and correctly sized, no automatic signal will see it. Only a human reading a sample, or a user complaining, will.
- Failures concentrated in a rare segment. A regression set sampled uniformly under-represents the tail by construction. If a specific customer or language is affected, aggregate rates can stay flat while that segment is entirely broken. Stratify the set deliberately.
- The judge drifting with the model. If your regression set is scored by another model, a provider-side change can move the scorer and the scored together. Judge bias is the reason a handful of the archived cases should carry human-written expected answers that never change.