Code Review Prompts That Find Real Problems

An AI reviewer that comments on everything gets muted in a fortnight. The fix is not a shorter prompt or a politer tone: it is a severity rubric with a floor, a required field that a speculative finding cannot fill, and permission to return nothing.

The prompt

Review the change in <diff>. <context> contains the full current contents of
every file the diff touches.

Report only findings that meet a level in <severity>. Findings about naming,
formatting, import order, comment wording, docstring style, type-annotation
style, or any preference that does not change behaviour are OUT OF SCOPE and
must not be reported, however confident you are that they would improve the
code. The linter and the formatter own those.

<severity>
blocker  Data loss, a security hole, money or quantities computed wrongly, or
         a crash on an input this code will actually receive in production.
         You must name the input.
major    Correct on the happy path and wrong on a case this code will meet: an
         unhandled error from a call that can fail, a race between two
         concurrent callers, an unbounded query or loop, an off-by-one at a
         boundary, a resource never released.
minor    Correct today, will cause an avoidable incident later: a network call
         with no timeout, an exception swallowed without a log, a log line
         that will print a secret or personal data, a test that cannot fail,
         a migration with no rollback.
</severity>

For each finding return:
{"severity": "blocker" | "major" | "minor",
 "file": "...", "line": <number in the new file>,
 "claim": "<what goes wrong, one sentence>",
 "trigger": "<the concrete input, sequence or interleaving that causes it>",
 "evidence": ["<verbatim lines from <context> that show it, with file:line>"],
 "fix": "<the smallest change, as a diff hunk>"}

A finding whose "trigger" you cannot state concretely is not a finding.
Delete it. "If the input were malformed" is not a trigger; "a row where
amount is the empty string, which the importer produces for cancelled
orders" is.

If nothing meets the bar, return {"findings": [], "unverified": [...]}.
An empty review is the correct output for most diffs. Do not lower the bar
to produce one.

Put in "unverified" anything you suspect but cannot check because the cause
is in a file not present in <context>. Give the file you would need and the
question you would ask of it. Do not report it as a finding.

<diff>{{diff}}</diff>
<context>{{full_files}}</context>

The severity rubric

The three levels are defined by consequence and by likelihood, not by category of defect, and that is the choice that makes them usable. A rubric organised by category — security, performance, correctness, maintainability — gives no guidance about whether a given finding belongs in the review, because every finding belongs to some category.

There is no nit level, and adding one is the change to resist. A nit level does not reduce nit comments; it legitimises them and they arrive in volume, which is precisely how automated review gets muted. The tuning question in general — signal-to-noise in automated review comments — is a page of its own; this prompt is one implementation of its conclusion.

The trigger field does the filtering

Every instruction to “only report significant issues” fails for the same reason: significance is a judgement the model makes about its own output, and it is generous with it. The trigger field replaces the judgement with a production task.

To fill trigger, the model must produce a specific input, call sequence or interleaving. A speculative finding cannot produce one without inventing it, and an invented trigger is visible to a reviewer in a way that a vague finding is not — you read “a row where amount is the empty string, which the importer produces for cancelled orders” and you can check whether the importer does that. This is the same device as the evidence span in classification and the quote requirement in extraction: replace an unverifiable judgement with a verifiable artefact.

The evidence array plays the second half of that role. It must quote lines from context, which is the full current file rather than the diff — because most real defects in a diff are about interaction with code the diff does not show. A reviewer given only the diff reports diff-shaped problems.

The standing ban on style findings

The ban is a whole paragraph rather than a bullet, it appears before the rubric rather than after it, and it names eight specific categories. All three choices are deliberate: the model has strong priors that a code review includes style commentary, and a short instruction competing with a strong prior loses.

The clause “however confident you are that they would improve the code” is there because the model’s style suggestions are often correct. Correctness is not the criterion; ownership is. A formatter that runs on commit makes a style comment pure cost, and a human reviewer whose attention is spent on import order has less left for the unbounded query.

If a style rule genuinely matters to you and no tool enforces it, the answer is to make a tool enforce it, not to add it back to the review prompt. That keeps the review channel for things a tool cannot decide.

Why an empty review must be allowed

Instruction-tuned models are shaped to produce output. Asked to review a diff, a model with nothing to say will find something, because returning nothing reads as failing the task. The result is that your low-severity findings are dominated by the diffs that were fine.

Two sentences fix it, and both are needed. “An empty review is the correct output for most diffs” reframes the task so that silence is success. “Do not lower the bar to produce one” blocks the specific move of downgrading a finding into minor to have something to say.

The unverified array gives the model a legitimate place to put genuine uncertainty. Without it, suspicions become findings, because there is nowhere else for them to go. With it, you get a useful secondary output: a list of files your review context is missing, which tells you how to assemble a better context next time.

One diff, two candidate findings

--- a/billing/refund.py
+++ b/billing/refund.py
@@
-def refund(order_id: str, amount_minor: int) -> Refund:
-    order = db.get_order(order_id)
+def refund(order_id: str, amount_minor: int, reason: str = "") -> Refund:
+    order = db.get_order(order_id)
+    already = sum(r.amount_minor for r in db.refunds_for(order_id))
     if amount_minor > order.total_minor:
         raise TooLarge(order_id)
-    return psp.refund(order.psp_ref, amount_minor)
+    return psp.refund(order.psp_ref, amount_minor, idempotency_key=order_id)

Two things a reviewer might say about that diff. One of them is a finding under this rubric and one is not, and telling them apart is the whole exercise.

{"findings": [
  {"severity": "blocker",
   "file": "billing/refund.py", "line": 7,
   "claim": "Partial refunds can exceed the order total in aggregate.",
   "trigger": "Two 60.00 refunds against an order with total 100.00. Each
               passes the amount_minor > order.total_minor check
               independently; 'already' is computed and never used.",
   "evidence": ["billing/refund.py:5  already = sum(r.amount_minor for r in db.refunds_for(order_id))",
                "billing/refund.py:7  if amount_minor > order.total_minor:"],
   "fix": "-    if amount_minor > order.total_minor:\n+    if already + amount_minor > order.total_minor:"},

  {"severity": "major",
   "file": "billing/refund.py", "line": 9,
   "claim": "The idempotency key is the order id, so a second legitimate
             partial refund on the same order is silently deduplicated by the
             PSP and returns the first refund.",
   "trigger": "A customer refunded 20.00 on Monday and 30.00 on Tuesday for
               the same order.",
   "evidence": ["billing/refund.py:9  idempotency_key=order_id"],
   "fix": "-    idempotency_key=order_id\n+    idempotency_key=f\"{order_id}:{refund_request_id}\""}],
 "unverified": [
  {"suspicion": "db.refunds_for may exclude refunds still settling, which
                 would make the aggregate check above wrong in the other
                 direction.",
   "need": "billing/repository.py",
   "question": "Does refunds_for filter on settlement status?"}]}

Both findings name a trigger you can act on: two specific refunds against one order. Neither is a style comment, and neither could have been written from the diff alone — already being computed and unused is visible in the diff, but understanding that refunds_for exists and what it returns needs the file.

What the rubric excludes here is just as instructive. The new reason parameter has a mutable-looking default, no docstring and is never used — three comments a reviewer would make and none of which changes behaviour. Under this prompt they do not appear, which is why the two that matter are read.

The unverified entry is the shape to aim for: a specific suspicion, the file that would settle it, and the question to ask of that file. It is directly actionable — add that file to the context and re-run — in a way that “there may be issues elsewhere” is not.

When it stops working

• The findings-per-diff rate rises without the code changing. Track it weekly. A rise is almost always the bar softening rather than the code getting worse, and the usual cause is somebody adding an example to the minor list.
• Triggers become generic. Sample ten and read them. “If the input is invalid” appearing repeatedly means the field has stopped filtering and become a formality. Restate the counter-example in the prompt.
• Nobody acts on the findings. The measurable version of muting. Log which findings led to a change in the same pull request; a level whose acted-upon rate falls below about a third is not earning its place, and the fix is to narrow its definition rather than to remove the level.
• unverified is always empty. Means the model believes it can see everything, which is rarely true for a diff in a real codebase. Check that context is actually being populated.