Data Versioning for Reproducible AI

“Can you reproduce the March evaluation?” is a question with a yes-or-no answer, and which one you get was decided in March. The good news is that most of what it takes is a file.

Everything a result depends on

Write out the full dependency set once, because the omissions are always the same three or four things.

It is worth separating two goals that get conflated here, because they need different amounts of work. Reproducing a result means getting the same number again from the same inputs. Explaining a result means being able to say what produced it — which document was retrieved, which prompt was assembled, what the model returned. The second is usually the one somebody actually wants, it is cheaper, and it is satisfied by logging rather than by pinning. Decide which one you are buying before you build for it.

The two you cannot pin

Two entries in that table are outside your control, and pretending otherwise is how a reproduction attempt produces a wrong answer instead of an error.

The hosted model

A provider can update a model behind a stable name, and dated aliases are retired on a schedule. So a run from six months ago may be impossible to reproduce exactly, not because you failed to record anything but because the function you called no longer exists. Silent model updates are the general form of this problem.

What you can do is record enough to detect it: the exact model identifier as the provider returned it in the response, not the string you sent; the response fingerprint field where one is offered; and the date. Then a failed reproduction says “the model identifier differs” rather than “the numbers moved and nobody knows why”.

Sampling

Even at temperature 0 with a fixed seed, hosted inference is not guaranteed to be bit-identical across runs — batching and kernel selection vary with load, and floating-point addition is not associative, so the same request can produce a different token where two candidates are nearly tied. Why temperature 0 is not deterministic is worth understanding before you promise anyone exact reproduction.

The practical consequence: define reproduction as “the metric falls within the confidence interval”, not “the outputs are identical”. That means recording the interval at the time, which means recording the per-item results and not only the aggregate.

How the tools version data

All of them do the same trick — content-address the data, keep pointers in something small — and differ in where the pointers live.

• DVC puts a hash-bearing pointer file in git next to the code, with the bytes in object storage. Reproduction is git checkout plus dvc pull, which is a genuinely nice property: one commit identifies code and data together.
• Table formats — Delta Lake and Apache Iceberg — version at the table level with a transaction log, giving you snapshot ids and time-travel queries (VERSION AS OF) without copying data. The right fit when the corpus is a table rather than a directory of files.
• Object versioning in the storage layer, plus a manifest listing the exact versions. Fewest moving parts, and it is all a document pipeline usually needs given that its artefacts are already content-addressed.

If you took the artefact-store advice from the pipeline overview, you have most of this already: keys are hashes, so nothing is ever overwritten and every past state is still addressable. The remaining work is recording which set of keys constituted a run.

The manifest

One JSON file per run, written at the start and finalised at the end, stored with the results. Not a database — a file, so it survives the database.

{
  "run_id": "2026-03-14T09:12:04Z-eval-retrieval",
  "code": {
    "commit": "9c1f0b2e5a...",
    "dirty": false,
    "python": "3.12.4",
    "packages_sha256": "b41d..."          # hash of the lockfile
  },
  "config": {"chunk_size": 800, "overlap": 100, "top_k": 8},
  "corpus": {
    "snapshot": "corpus/2026-03-13",
    "document_count": 204118,
    "manifest_sha256": "77ae..."          # hash of the sorted id+hash list
  },
  "models": {
    "embedding": {"requested": "text-embedding-3-small",
                  "returned":  "text-embedding-3-small"},
    "generation": {"requested": "vendor/model-x",
                   "returned":  "vendor/model-x-2026-02-11",
                   "temperature": 0, "seed": 7}
  },
  "eval_set": {"name": "retrieval-golden", "version": 4, "items": 312},
  "results": {"recall_at_8": 0.81, "ci95": [0.77, 0.85],
              "per_item": "results/2026-03-14/items.jsonl"}
}

Two fields do most of the work. corpus.manifest_sha256 is the hash of a sorted list of document ids and their text hashes — one number that answers “is this the same corpus?” without comparing two hundred thousand rows. And models.generation.returned next to requested is what makes a provider-side model change visible as a diff rather than as a mystery.

Make the reproduction script read the manifest and refuse to run on a mismatch, listing every field that differs. A reproduction that silently proceeds against a different corpus is worse than no reproduction, because it produces a number that looks comparable.

What reproducibility buys day to day

The audit story — “prove why the system said that in March” — is the one used to justify the work, and it is not the one that pays for it. Three ordinary things do.

Bisection: when quality drops and three things changed, snapshots let you vary one at a time. Fair comparison: a new chunking strategy evaluated against a corpus snapshot rather than against a corpus that has grown since. And onboarding: a new engineer who can reproduce last quarter’s number exactly has a working system and a baseline, rather than a repository and a Slack thread.

Set a retention policy on the snapshots, though, or this becomes the largest line on the storage bill. A corpus snapshot for every nightly run is almost always more than anyone needs. Keeping every run for a fortnight, one run a week for a quarter, and one a quarter indefinitely covers bisection, seasonal comparison and audit at a small fraction of the cost — and because the artefacts are content-addressed, snapshots that share documents share storage rather than copying it.