Reviewing a Snapshot Diff Before Approving It

Approving a golden-file diff is a decision made in about two minutes, dozens of times a week, by somebody who mostly expects it to be fine. The way to make that decision reliable is not more care; it is a fixed order of checks that puts the dangerous changes before the boring ones.

Read the shape of the change set first

Before opening a single file, look at the counts. How many cases changed, out of how many? Are they clustered in one area of the corpus or spread evenly across all of it? Did any files get added or deleted?

These answer a question the individual diffs cannot. A change confined to the eleven refund cases is consistent with an edit to the refund clause and can be reviewed as such. A change spread evenly across the entire corpus is almost never a content change — it is a formatting change, a serializer change, a normalisation change or a model change, and reviewing it case by case will teach you nothing that reviewing one case would not. Deleted files are the highest-risk item in any golden-file diff and the easiest to miss: a deleted golden file means a test stopped existing, and a test that stopped existing never fails again.

The per-case checklist, in order

Once you open a case, run these in this order and stop at the first one that gives you a reason to reject. The order is deliberate: the early checks are cheap and catch the failures with consequences, and the late ones are the judgement calls that consume attention.

• Did a number change? Days, amounts, percentages, quantities, identifiers. A number is the thing most likely to be acted on by a human reading the answer and the thing least likely to have changed for a harmless reason. Never approve a changed number without knowing which is right.
• Did the finish reason change? A move from a normal stop to a length stop means the answer is truncated, and truncated answers often still read as complete for the first two paragraphs.
• Did the tool calls change? A different tool, a different order, or — most importantly — no tool where there used to be one. A model that stopped calling the lookup and started answering from memory produces fluent, confident, unverified text, and this is the single most expensive regression on the list.
• Did a refusal or safety message appear or disappear? Both directions matter and both are usually unintentional.
• Did a negation flip? “You may return” versus “you may not return” is one token in the diff and the entire meaning of the answer.
• Did something appear that was not in the context? A new fact, a new citation, a new policy name. Additions are more dangerous than removals, because a removal is visible to the end user and a fabrication is not.
• Only then: is the rewording acceptable? If the first six are clean, this is usually yes, and it is the question people start with.

Tells that a diff is not what it looks like

A few patterns mean the diff is misleading and should send you somewhere other than the approve button. If every case changed by exactly the same transformation — a quote style, an indentation, a key order — the change is in your serializer or normaliser, not in the model, and the correct review is of that code rather than of four hundred files.

If the served model identifier changed in any case, review nothing else until that is explained: an alias moved, a fallback fired, or a test profile is not pinned, and every other difference in the change set is now unattributable. If a case’s diff is enormous while its neighbours are small, the likely cause is an early token divergence rather than a large behavioural change, which is a property of how autoregressive generation amplifies one different token and a sign that this case should be recording a structure rather than prose.

And if you cannot tell what changed at all, the artefact is wrong rather than you. Say so on the pull request rather than approving it; “this diff is not reviewable” is a legitimate and useful review outcome, and it is the only feedback that ever gets golden files reduced to something a person can read.

Reviewer attention is the scarce resource

Every design decision in a golden-file suite should be evaluated against how many cases a person will genuinely read. That number is small — small enough that a corpus which regenerates wholesale is effectively unreviewed no matter what the process document says.

Three things buy attention back. Order the diff so that cases matching the high-risk checks above appear first, rather than alphabetically. Keep a small tier of cases that are always reviewed by a human and a large tier checked only by machine assertions, and be explicit about which is which. And make the artefact small: a golden file of six fields is read, a golden file of six paragraphs is skimmed. The reviewing workflow and the approval it produces are only worth as much as the reading that actually happens.

It follows that the honest response to a change set nobody can review is to change the suite rather than to review harder. If the same twenty cases produce a wording diff on every regeneration, they are recording the sampler and should be recording a structure. If a tier of the corpus is approved without being read every time, stop pretending it is reviewed and give it machine assertions instead, so that its failures mean something. Both moves shrink the thing a person has to look at, which is the only lever that reliably improves the quality of these decisions.