Golden Master Testing for a Legacy Prompt You're Afraid to Touch

Michael Feathers’ term for this is a characterization test: one that documents what the code currently does rather than what it should do. It is exactly the right tool for an eight-hundred-word prompt nobody wants to touch — with one adaptation, because the output is not reproducible.

Why you cannot snapshot the output

The classic move is to record the output for a set of inputs, commit it, and treat any change as a failure. Applied to a model call it produces a suite that fails on the first rerun, because temperature zero reduces variance without eliminating it: batching, hardware, floating-point non-associativity and provider-side updates all move the text. A team that meets this usually responds by loosening the comparison — fuzzy matching, similarity thresholds — and ends up with a suite that fails for reasons nobody can characterise and passes over changes that matter.

The adaptation is to golden something else. The output is not stable, but a projection of it usually is, and the projection is where the behaviour you are afraid of losing actually lives.

Golden the projection instead

A projection is a deterministic function of the output that discards the prose. Candidates, roughly in order of how reliably they hold:

• Shape. The sorted set of keys in a JSON output, and each value’s type. Nearly always stable, and the thing most likely to break when a prompt is edited.
• Classification. Which of a fixed set of labels was chosen. Stable when the categories are well separated, and precisely the behaviour that regresses when someone reorders the examples.
• Tool calls. The names called and the argument keys supplied, in order. Values often vary; names rarely do.
• Entity recall. Which identifiers from the input appear in the output — an order number, a currency, a date. This is the projection that catches a prompt edit which quietly stopped carrying a required fact.
• Numbers, within a tolerance. An extracted total is either right or wrong; comparing to two decimal places is a real assertion, not a fuzzy one.
• Bands. Output length bucketed coarsely, language detected, whether it refused. Weak individually, useful in aggregate.

Serialise the projection, not the text, and the master becomes a file a reviewer can read. That is worth as much as the test: the projection written out for two hundred inputs is frequently the first written description of what the prompt does.

Discovering what is stable

Do not decide the projection by intuition. Measure it, before changing anything:

1. Sample 150 to 300 real inputs from logs, redacted, spread across the segments the prompt actually sees rather than the ones you remember.
2. Run each input three to five times against the current prompt, with the model id, parameters and SDK version all pinned and recorded.
3. Compute every candidate projection on every sample.
4. Keep only the projections that agree across all samples for an input. A field that varies between runs today was never a guarantee, and freezing it manufactures a flaky test out of thin air.
5. Write the survivors to the master file along with the pinned configuration and the date.

Step four is the one that makes this work. It converts “what do I think this prompt guarantees” into “what does it demonstrably do repeatedly”, and the gap between those two is usually where the fear came from. Record the inputs whose projections disagreed too: those are the cases where the prompt is genuinely unstable, and knowing which they are is a finding, not a gap.

# master.jsonl — one line per input, plus a header record
{"_config": {"model": "…", "temperature": 0, "sdk": "…", "captured": "2026-08-11", "samples": 5}}
{"input_id": "inv-0031", "keys": ["amount","currency","due_date","order_id"],
 "label": "overdue", "tools": ["lookup_invoice"],
 "entities": ["4417","EUR"], "amount": 128.40, "length_band": "short"}

A diff is a decision, not a failure

Once the master exists, editing the prompt produces a list of changed projections. Resist wiring that straight to a red build. The output of a golden master run is a review artefact: nine inputs changed label, two stopped calling a tool, one gained a key. Some of those are the change you intended and some are not, and only a person can say which.

Practically, that means the master run posts a summary and requires an explicit accept, committed in the same change as the prompt edit. The diff and the reasoning end up in the same review, which is the artefact you want six months later when someone asks why the label for partial refunds moved. It also means the master is regenerated deliberately rather than by a bot with an update flag, which is how snapshot suites usually decay into files nobody reads.

A useful summary counts changes per projection rather than per input: “label changed on 9 of 220, keys unchanged, tool calls changed on 2” is triage. Two hundred and twenty individual diffs is not.

What makes a master expire

A master is only valid for the configuration that produced it, which is why the header record above exists. Four things invalidate it, and each should force a recapture rather than a comparison:

• A different model or model version, including an alias that silently moved (silent model updates).
• Different sampling parameters. A temperature change invalidates the stability measurement the master was built on.
• A different environment. Comparing a master captured against staging with a run against production is comparing two unknowns (staging config mirroring the production model).
• Age. Set an expiry — a quarter is reasonable — after which the master is recaptured against the current model even with no prompt change. The recapture diff is itself the measurement of how much the provider moved under you.

None of this makes the prompt safe to refactor. It makes the consequences of refactoring visible, which is the actual goal: the fear of touching a legacy prompt is fear of an unobservable change, and a master converts it into a list you can read in ten minutes. Once the list is boring, the refactor is ordinary work, and the same corpus becomes the basis for the golden dataset the rest of the suite uses.