A Code Review Checklist for a Prompt Change Pull Request

A prompt change is a behaviour change to every request that uses it, shipped through a review process designed for code whose behaviour you can read off the diff. The reviewer’s problem is that a six-word edit can move accuracy, cost, latency and output format at once, and none of that is visible in the patch.

Why a prompt diff reviews badly

Ordinary code review works because the diff is close to the effect. Change a comparison operator and a reviewer can trace what happens. Prompt edits break that link in three ways.

• The effect is distributional. Adding “be concise” does not make the output concise; it shifts a distribution over outputs. Whether that helps is a question about many samples, and nobody can answer it by reading.
• The blast radius is undeclared. A shared system prompt may serve six features. The diff shows one file; the change reaches everything that imports it, including whichever downstream parser depends on a format the edit just relaxed.
• Small edits are not small. Reordering instructions, changing an example, or moving a constraint from the end to the middle can all change behaviour more than adding a paragraph. Position in the prompt is not neutral — see prompt sensitivity.

So the review cannot be “does this text look reasonable”. It has to be a check that the author has produced the evidence a reader cannot derive.

The checklist

1. Is the diff readable at all? If the change appears as one modified line of a Python or TypeScript string literal, stop and ask for the prompt to be moved to a file. You cannot review what you cannot see word by word. This is worth doing once, permanently — storing prompts as files fixes it for every future PR.
2. Are all the consumers listed? The description should name every feature and every code path that reads this prompt. If the author does not know, that is the finding: a prompt with unknown consumers cannot be safely edited.
3. Did the regression suite run, and on what? Not “tests pass” — the unit tests were always going to pass. You want the golden-dataset run, its pass rate before and after, and the identifiers of any case that changed verdict in either direction. A case that started passing is as much of a question as one that stopped, because it may have started passing for the wrong reason.
4. Do the output-format guarantees still hold? If anything downstream parses the response, the schema assertions must still pass. Edits that add prose instructions frequently loosen format compliance as a side effect.
5. Are the template variables still correct? Every placeholder in the new text must be supplied by the calling code, and every supplied variable should still be used. An unsubstituted placeholder reaching the model is a silent quality bug rather than an error.
6. What is the token delta? One number, measured with the tokenizer the provider documents, on a representative rendered prompt. See the next section for what to do with it.
7. Does the version identifier change? If you record a prompt hash on responses — and you should, so that a support ticket can be traced back to the prompt that produced it — then this PR must produce a new hash, and nothing else should have to be updated by hand for that to happen.
8. Is there a rollback path that is not a revert-and-deploy? Ask how this gets turned off in two minutes at 2 a.m. If the answer is “revert the PR and redeploy”, that is a real risk to weigh against the size of the change.
9. Are safety-relevant instructions intact? Refusal behaviour, PII handling, and any instruction that exists because of a past incident. These get deleted by accident during rewrites-for-clarity more often than by intent, so check for removals specifically.

The cost delta, computed

Token deltas on a system prompt multiply by every request, which makes them easy to underestimate from the diff. Compute it explicitly, with each input named.

Take a change that adds 180 tokens to a system prompt on an endpoint serving 400,000 requests a month, at a hypothetical input price of $3.00 per million tokens. Then 180 × 400,000 = 72,000,000 extra input tokens a month, which at $3.00 per million is $216 a month. Every figure there is an assumption stated in the sentence that uses it: substitute your own token count, your own volume, and the price your provider lists on the day you read it. The arithmetic is the point, not the total.

Two adjustments usually apply. If the added text sits inside a cached prefix and your provider bills cached input at a reduced rate, the effective cost is lower — but only for requests that hit the cache, so you need the hit rate to say by how much. And if the edit changes output length, the output delta usually dominates, because output tokens are typically priced several times higher than input. Measure the mean output length on the regression run rather than guessing at it.

Evidence the PR should carry

Make this a template so it is not a negotiation each time. A prompt change PR should include the rendered before-and-after prompt for one real input; the regression run identifier with pass rates on both sides and a list of flipped cases; the token delta and its cost arithmetic; the list of consumers; and the rollback mechanism. Five items, all of which the author already has or should have got before opening the PR.

The rendered prompt matters more than it sounds. Reviewers read the template; the model reads the rendered result, complete with whatever the retrieval step injected and whatever whitespace the templating engine produced. Bugs that live in that gap — a stray delimiter, a doubled newline that separates an instruction from its example — are invisible in the template diff and obvious in the render.

When to block and when to let it ship behind a flag

Blocking a prompt change until it is proven correct is a good way to stop anyone improving prompts, because the proof is expensive and partial. The more workable line: block on evidence, not on outcome.

Block when the PR has no regression evidence at all, when a schema assertion regressed, when a safety instruction was removed without being mentioned, or when there is no rollback path faster than a deploy. Approve with a smaller change — behind a flag, at a fraction of traffic — when the evidence exists but is ambiguous: a small pass-rate movement inside the noise of your suite is exactly the case a canary settles and a reviewer cannot. Deciding what fraction and what to compare against is its own question, covered in choosing a canary percentage.

The last reviewer habit worth building: ask what the author expected to happen, before reading the results. A prompt change with a stated hypothesis is reviewable. One that is simply “this seemed better” has no failure condition, which means the canary that follows it has nothing to fail.