Dataset Documentation: Datasheets That Get Used

Every dataset ends up with a new owner, and the questions they ask are remarkably consistent. A datasheet is worth writing exactly to the extent that it answers those questions and no further.

The questions a datasheet has to answer

Not hypothetical. These are what someone asks when they inherit a dataset, in roughly the order they ask them:

• Can I use this for what I want to use it for, legally?
• Where did the rows come from, and does that include anything generated?
• Who or what produced the labels, and how much do they disagree?
• What was thrown away before I got it, and on what rule?
• Is any part of this contaminated with the eval set I am about to use?
• Does it contain personal data, and if so under what basis?
• What is it bad at — which slice is thin or missing entirely?
• Is it still current, and who do I ask?

A document that answers those eight is worth more than a fifty-field form nobody completed. That is the design constraint for everything below.

It is worth being concrete about what the absence costs, because documentation is the first thing cut and the cost is always paid by somebody else. A dataset with no provenance record cannot be cleared for a new use, so it gets rebuilt. A dataset whose filters were never written down cannot be extended consistently, so the new rows are filtered differently and nobody notices for a year. A dataset with no record of what it was decontaminated against gets used to evaluate a model it was partly trained on. And a dataset whose known gaps live only in the head of the person who built it becomes, the day they change teams, a set of numbers with no error bars.

Each of those is a week or more of work, spent later, by someone with less context. The datasheet is not a governance artefact bolted on for appearances — it is the difference between an asset and a liability with rows in it.

The published frameworks

Use the full Gebru questionnaire when the dataset is being published externally, when a regulator or an ethics board will read it, or when it involves human subjects. For an internal training set, the long form is why the documentation does not exist. Cut it.

The judgement to apply when cutting: keep every question whose answer would change what a future user does, and drop every question whose answer is only interesting. “What was the motivation for creating this dataset” changes behaviour, because it tells someone whether their intended use is in or out of scope. “Who funded the creation of the dataset” usually does not, internally, and it is the sort of field that turns a living document into a form. The external version of the same datasheet can have both; the version people actually maintain should have the first.

A template short enough to be filled in

YAML rather than prose, deliberately: fields can be validated, diffed and required by CI, and a missing field is visible rather than rhetorically finessed. Keep it in the repository next to the data, not in a wiki.

name: support-intent-v4
owner: platform-ml            # a team, never a person who might leave
created: 2026-08-03
supersedes: support-intent-v3
status: active                # active | frozen | deprecated

purpose: >
  Fine-tuning the ticket router. NOT an evaluation set — see
  support-intent-eval-v2 for that.

composition:
  rows: 41320
  unit: one customer ticket + one intent label
  splits: {train: 37188, dev: 2066, test: 2066}
  split_by: customer_id       # NOT random: prevents leakage across splits

sources:                      # must sum to rows; generated data is declared
  - {name: zendesk-export, rows: 28110, span: "2024-01..2026-06", licence: internal}
  - {name: generated-gap-fill, rows: 9210, generator: "<model id>", licence: internal}
  - {name: partner-corpus, rows: 4000, licence: "CC BY-SA 4.0", attribution_required: true}

labels:
  taxonomy: docs/intents-v4.md
  produced_by: "2 annotators + adjudication; 9% double-labelled"
  agreement: {method: cohen_kappa, value: 0.71, n: 3719}
  known_confusions: ["billing vs subscription", "bug vs how-to"]

processing:
  deduplication: {method: minhash, shingle: 5, threshold: 0.8, removed: 3104}
  decontamination: {against: support-intent-eval-v2, method: 13-gram, removed: 87}
  filters: ["len(text) < 10 chars dropped (1210)", "non-EN/NL dropped (2933)"]

personal_data:
  present: true
  categories: [name, email, order_id]
  treatment: "emails and order ids replaced with stable pseudonyms; names retained"
  basis: legitimate_interest
  dpa: docs/dpia-support-ml.md

known_gaps:                   # the section future readers read first
  - "Enterprise tier under-represented (3% of rows, ~18% of revenue)"
  - "No tickets from the mobile app before 2025-04 (not logged)"
  - "Generated rows cover only the 6 thinnest intents; see gap-fill run notes"

maintenance:
  refresh: quarterly
  contact: "#ml-platform"
  retire_when: "taxonomy v5 ships"

The fields the generator writes for you

Half of that template should never be typed by a human, because typed numbers are wrong by the second revision. Emit them from the pipeline that builds the dataset:

• Row counts, split sizes and source breakdown. Computed, always. A hand-written row count is a claim about a previous version.
• Deduplication and decontamination results. Method, threshold and rows removed come straight out of the dedup pass. The interesting number is how many were removed, and only the pipeline knows it.
• Filter counts. Every filter reports what it dropped. A filter whose count nobody records is a silent deletion.
• Agreement statistics. Out of the annotation tool, as in preference collection.
• The generation manifest. Model identifier and version, prompt template hash, sampling parameters, run date, candidates generated versus kept. Without it, a generated slice is unreproducible and its provenance is a guess.

What a human must write is the part no pipeline can compute: the purpose, the taxonomy decisions, the known gaps, and the sentence saying what this dataset must not be used for. Those four are the document. Everything else is a report, and reports should be generated rather than typed — a number a person maintained by hand is a number that was accurate once.

Making it survive handover

• Version the datasheet with the data. Same repository, same commit, same tag. A datasheet in a wiki describes a dataset that no longer exists.
• Fail CI on a missing required field. The only mechanism that reliably keeps documentation current is a build that breaks without it. Validate the YAML against a schema and requireknown_gaps to be non-empty — every dataset has gaps, and an empty list means nobody looked.
• Make the gaps section a running log. Append to it every time the dataset surprises somebody. It is the section with the highest ratio of usefulness to effort in the whole file.
• Name a team, not a person. The single most common reason a datasheet becomes useless is that the contact left.
• Record what it must not be used for. The training set repurposed as an evaluation set is the most common and most damaging misuse of a dataset, and one line in the purpose field prevents it.