Updating Golden Files After an Intentional Prompt Change

You edited one clause of a system prompt and four hundred golden files now differ. Running the update flag makes the suite green in six seconds and destroys the only record of what the system used to do. The procedure below keeps that record.

What goes wrong in a bulk regeneration

A bulk update is a single action that accepts an unbounded number of changes at once, and it is the one moment in the whole workflow where a regression can enter the repository with a green build behind it. Three distinct things are mixed together in that four-hundred-file diff: the change you intended, the changes caused by sampling variance in the same run, and any change caused by something you did not intend at all — a truncated context, a tool that stopped firing, a provider that quietly moved the alias underneath you.

Nothing in the tooling separates them for you. jest -u, pytest --snapshot-update and their equivalents all mean “whatever happened is now correct”. The separation has to come from the order in which you do things.

The procedure

1. Isolate the change. Put the prompt edit on its own branch and commit it alone, with no golden files touched. The build is red at this point and that is correct; a red build is the measurement you are about to take.
2. Run the suite with no update flag and capture the failure list. Redirect the reporter to a file — a JSON reporter is easier to count than console output. This list is the blast radius, and it is the artefact the next step compares against.
3. Compare the blast radius to your prediction, which you wrote down before running anything. See the next section. Stop here if they disagree.
4. Regenerate. Run the update against the full suite, never against a filtered selection: several runners delete snapshots they consider unused, and a filtered run considers every unselected test’s snapshot unused.
5. Commit the regenerated files alone, as a second commit whose message names the prompt version that produced them. Reviewers can then read the two commits separately, and git log on the golden directory becomes a history of behavioural change rather than noise.
6. Run the suite twice more with no flags. Both runs must be green. Anything that fails now is not deterministic under your current configuration and never should have been an exact snapshot — see the flapping section below.
7. Review the golden-file commit case by case against a fixed checklist, and get the approval from somebody who did not write the prompt change.

Predict the blast radius first

The single most useful habit here costs thirty seconds: before running anything, write in the pull request description how many cases you expect to change and which ones. “This tightens the refund wording, so the eleven refund cases should move and nothing else.”

Then compare. If eleven cases move, you have learned something real and the review is small. If four hundred move, you have learned something more important: either the edit changed the output format globally rather than the behaviour locally, or your golden files are recording prose and are therefore recording sampling variance, in which case the honest fix is to reduce what you snapshot rather than to approve four hundred diffs. If three cases move and none of them is a refund case, stop entirely; something is wrong with your understanding of what the prompt does.

This works because a prediction made before the run cannot be rationalised after it. The same information is available afterwards and nobody uses it, because a diff you are already looking at always looks explicable.

Separating flapping cases from changed ones

A flapping golden file is one that differs between two consecutive runs with nothing changed. It is not a regression and it is not an intentional change; it is a case whose assertion was always wrong, and a bulk regeneration is exactly when it is cheapest to find.

Detect them by running the suite twice against the unchanged baseline before you regenerate, and recording which cases fail. Those cases carry no information about your prompt change, so exclude them from the blast-radius comparison and fix them separately: reduce them to a structure, or move them to an invariant assertion. Absorbing them into the regeneration is how a suite acquires a permanent background rate of meaningless diffs, and a permanent background rate of meaningless diffs is how review stops happening.

Guards that make this repeatable

• CI may never write a golden file. Enforce it structurally: run the tests, then fail the job if the working tree is dirty. A runner flag that refuses to write new snapshots is a good second layer, but the dirty-tree check catches every mechanism at once.
• Record the request, not just the response. Store the fully assembled prompt and parameters alongside each golden file. When a case moves unexpectedly, diffing the recorded requests answers in seconds a question that otherwise takes an afternoon.
• Pin the model to a dated version for the suite. An alias makes every regeneration ambiguous, because you can never rule out that the endpoint moved on the same day you did.
• Version the prompt, and put the version in the commit message. Prompt versioning is what makes “which prompt produced this file” a lookup rather than an archaeology exercise.