Reviewing AI-Written Code: A Checklist

The problem with reviewing generated code is not that it is bad. It is that it is fluent, idiomatic, well-named and commented, and every heuristic a reviewer has developed for spotting trouble was trained on code that looks worried.

Human review is calibrated on human error

A tired human writes an off-by-one, forgets a null check, leaves a debug print, names something badly. Reviewers are extremely good at this because the errors correlate with surface signals — inconsistent naming, a comment that trails off, a function that grew three limbs.

A sampler produces the highest-probability continuation. Its errors are the opposite shape: confident, conventional, symmetrical, and wrong at the level of premise rather than execution. The variable names are good. The comments are complete. The error handling is present. Nothing about the diff looks worried, and the reviewer’s attention allocation is driven by exactly the signals that have been removed.

The practical consequence is that reading more carefully does not work well, because the failure is not one of care. What works is checking specific things that fluency cannot fake.

The eight classes, worst first

Ordered by how invisible each one is in a unified diff — the top of this list is what a green CI run and a nodding reviewer both miss.

1. Semantics changed inside a refactor

The task was “extract this into a helper” and along the way a >= became a >, an early return moved, or a caught exception is now logged instead of rethrown. This is the worst class because the diff is large and mostly mechanical, so attention is spent confirming the mechanical part. Never review a refactor for correctness; make it prove itself instead — see the codemod-and-ratchet approach.

2. A missing check that was never in the diff

Nothing was deleted, so there is nothing to notice. The new endpoint loads by id from the path and never asks whether the caller owns the row. Absence is invisible in a diff by construction, which is why this class needs a checklist rather than a reader.

3. Invented API surface

A method that ought to exist and does not: path.existsOrCreate(), a timeout option on a client that takes signal, a keyword argument added in a later major version. Cheap to catch — the type checker or the import resolver finds most of it — but untyped languages and dynamic dispatch let it through to runtime.

4. Code written against the wrong version

Compiles, runs, subtly different behaviour. Pandas, React and every HTTP client have deprecation-shaped versions of this. Check any unfamiliar call against the version in the lockfile, not against memory.

5. Copy-consistent duplication

The model reproduced the neighbouring pattern instead of calling it, because the neighbour was in the context and the shared helper was not. Each instance is defensible; the aggregate is the debt.

6. A test that asserts the mock

Stub returns 3, assertion checks 3. Coverage rises, information content is zero. Covered in full on the test generation page.

7. Error handling that is present but not exercised

A try/except around the wrong line, a retry that retries a non-idempotent write, a fallback branch that has never executed and does not compile-check its own return type in a dynamic language.

8. Comments describing intent the code does not have

The comment says it validates the input. It does not. This one is last because it is comparatively harmless on its own — but it poisons every later reading of the file, including the next model’s.

Two moves that beat reading harder

Run the new test against the old code

If a change is supposed to fix a bug and ships a test, the test must fail on the parent commit. If it passes, the test does not test the fix, and you have just learned that with one command rather than in three months:

git stash                       # or: git worktree add ../old HEAD~1
git checkout HEAD~1 -- src/     # old source, new tests
pytest tests/test_invoice.py -x
# expected: FAILED test_invoice.py::test_locked_period_rejected
# if it PASSES, the test proves nothing about the change

The same move generalises: any test added alongside a behaviour change should fail without the change. It is the single cheapest check available and almost nobody runs it.

Read the deletions first

Filter the diff to removed lines and ask what replaced each one. Class 1 lives entirely in the deletions, and reading a diff top-to-bottom buries them among additions:

git diff main... -U0 | grep '^-' | grep -v '^---'

For a refactor that is supposed to preserve behaviour, a stronger version exists: normalise whitespace and compare the two versions symbol by symbol, so that anything that is not pure movement stands out. git diff --color-moved=zebra -w gets most of the way there for free.

There is a third move worth knowing for anything high-stakes: run the same task twice from the same starting state, in two fresh sessions, and diff the two results against each other. Where the two agree, the answer was determined by the code and the task. Where they diverge, the model was choosing freely — an invented error message, a guessed default, an interface it had to make up — and divergence is a precise map of the decisions nobody specified. It costs one extra generation and it points your attention at exactly the lines where the code is an assumption rather than a consequence.

What to change about the process

• Require the author to have read it. Not a policy statement — a description in the PR of what the change does and why, written by the human. Generated descriptions of generated code close the loop with nobody in it.
• Ask the model for its own uncertainty list before you read. “List the assumptions you made that you could not verify from the code shown, and the call sites you did not check.” It is not reliable, but it is free and it points at classes 2 and 4 more often than chance.
• Size the diff. Review quality falls off a cliff with diff size and generated diffs are large by default. A 400-line change that could have been four 100-line changes is a process failure, not a review failure.
• Never let automated review substitute for this. A bot is a filter for a few classes, not a reviewer; what it can and cannot be tuned to do is a page of its own.