Golden Datasets: Building and Maintaining One

Nearly every team that builds a golden dataset builds a good one. The problem is eighteen months later, when the person who wrote it has moved teams and nobody remaining can say why item 47 is graded the way it is — so nobody dares change it, and nobody trusts it either.

How golden sets actually die

Not through neglect, usually. Through four specific losses, each of which happens while the file itself looks perfectly healthy:

• Lost intent. An item’s expected output encodes a policy decision — the refund window is 30 days, we never promise a callback time — and the policy changed. The item is now testing an obsolete rule and reads as a model failure.
• Lost provenance. Nobody knows whether item 47 came from a real incident, from a brainstorm, or from a model that generated it. These deserve very different amounts of respect and the file does not distinguish them.
• Silent saturation. Every candidate scores 96% and the set has stopped discriminating, but the number still looks healthy on a dashboard, so nobody notices that the instrument has no resolution left.
• Unresolved disputes. Two engineers disagreed about an item, one quietly changed it, the scores shifted, and a later comparison across that boundary is meaningless. Nothing recorded that this happened.

Every rule below exists to make one of those four structurally impossible rather than a matter of somebody remembering.

The schema does the remembering

One JSONL file, one item per line, with fields that carry the context a newcomer needs. The input and the expected output are the smaller half of it.

{
  "id": "gold-0147",
  "status": "accepted",
  "stratum": "should-refuse",
  "input": { "ticket": "...", "retrieved_docs": ["doc-88", "doc-12"] },
  "expected": {
    "kind": "rubric",
    "criteria": [
      "does not state a refund amount",
      "names the escalation path",
      "asks for the order number"
    ]
  },
  "why_this_exists": "INC-2291. Model promised a full refund on a
     partially-shipped order; finance had to honour it. This item is the
     regression test for that class, not for the wording.",
  "provenance": { "source": "incident", "ref": "INC-2291",
                  "captured": "2026-03-14", "captured_by": "r.okafor" },
  "policy_refs": ["refunds-policy@v7"],
  "reviewers": ["r.okafor", "s.mehta"],
  "last_reviewed": "2026-06-02",
  "disputes": []
}

Three of those fields are the ones people skip and the ones that matter most. why_this_exists is written in prose, at capture time, by the person who was there — it is the single highest-value field in the file and takes ninety seconds to write. Its rule is that it must say what class of failure the item guards against, not what the correct answer is; the correct answer is already elsewhere and restating it teaches nobody anything.

policy_refs pins the item to the version of an external rule it depends on. When the refunds policy goes to v8, a grep gives you every item that needs re-reading, which converts the lost-intent failure from an archaeology problem into a list.

provenance.source should be a small closed vocabulary — incident, traffic, authored, synthetic, adversarial — because those tiers earn different treatment. An incident-derived item is close to sacred. A synthetic item can be deleted by one person on a hunch.

On size: aim for enough items per stratum that the stratum can move independently, which in practice means twenty at the absolute minimum and forty to eighty where you intend to report a per-stratum number. A stratum of six items reports a score that jumps by 17 points when one item flips, and someone will read that jump as a finding. It is better to have four well-populated strata than nine token ones, and the strata you cannot populate are usually telling you that the failure class is not distinct enough to be worth tracking separately.

Item lifecycle

Items are not added and deleted; they move through states, and the states are in the file.

Deprecation deserves the emphasis. The instinct when an item becomes obsolete is to remove the line, which silently changes the denominator and makes every previous run incomparable. Marking it deprecated with deprecated_after: "v12" lets the harness reconstruct any historical scoring set exactly.

Review rules

• Two reviewers to accept, and they must not discuss it first. Independent agreement is evidence that the expected output is unambiguous. Agreement reached after a five-minute conversation is evidence of nothing except that one person was more persuasive.
• Anyone may raise a dispute; nobody may resolve their own. This is the rule that survives turnover, because it means no single person’s judgement is load-bearing.
• Each stratum has a named owner. Not to gatekeep, but so that the quarterly re-read is somebody’s job. Ownerless strata are the ones that saturate unnoticed.
• Quarterly: read the ten items with the highest pass rate across all candidates. These are your saturated items. If everything passes them, they are costing you money and telling you nothing — deprecate or harden them. This one habit is the whole defence against silent saturation.
• A new hire’s first task is to grade twenty items cold. Where they disagree with the file, either the file is ambiguous or the onboarding is. Both are worth finding, and this is the cheapest test of whether the set still explains itself.

Versioning and comparability

Tag the file on every accepted change and record the tag in every result row. Scores are only comparable within a version, and the single most common way an eval programme loses credibility is a chart with a step in it that turns out to be a dataset edit rather than a model change.

Ids are immutable and never reused. If an item’s expected output changes materially, that is a new id and a deprecation of the old one, not an edit — because an edit makes two different questions share a name, and every historical comparison involving that id becomes quietly wrong.

The cadence that makes all of this affordable is smaller than it sounds. Capture is continuous and costs a minute per incident. Review is a thirty-minute session a week in which two people clear the proposed queue and the disputes. The quarterly saturation pass is an hour. That is the entire maintenance budget for an artefact that becomes the most valuable thing your team owns about its own quality — and the reason it usually does not get spent is that nobody named an owner, not that anybody found it expensive. This is the same discipline that makes a CI gate over the set interpretable at all.