Reproducibility in AI Research

“Nobody can reproduce these results” is said often enough to have stopped meaning anything. It covers at least four distinct failures, with different causes and different implications for whether you should believe the paper.

Four things called reproduction

Being precise about the level changes the conversation. A result that reruns but does not survive a seed change is fragile. A result that cannot be reimplemented from the paper is under-described. A result that reproduces perfectly and does not generalise is fine and was simply over-read. These deserve different reactions and usually get the same one.

Why it fails here specifically

• Stochasticity everywhere. Initialisation, data order, dropout, sampling at inference, and non-deterministic kernel scheduling on accelerators. Two runs of identical code on identical hardware can differ, so “the same number” is already a question of tolerance rather than equality — related to why temperature zero is not deterministic in production either.
• Hyperparameter search is invisible effort. A method tuned across hundreds of configurations, compared against a baseline tuned across ten, will win. Nothing was falsified; the comparison was of effort. The search budget per arm belongs in every paper and appears in a minority.
• The data is the method. Corpus composition, deduplication, filtering and mixture ratios frequently matter more than the architecture change being proposed, and are the parts least likely to be released.
• Compute is a barrier. Even a fully specified training run may cost more than the replicator has. That is not concealment, but the effect on verifiability is identical.
• Publication pressure runs one way. A negative replication is hard to publish and career-neutral at best, so the checking work is systematically under-supplied relative to the producing work.

What replication attempts have found

There is no honest single number for how much of machine learning replicates, and anyone offering one should be asked how they counted. It has not been counted here. What does exist is a set of careful re-examinations of specific subfields, and the pattern across them is consistent enough to be worth describing.

The recurring finding is that reported improvements shrink or vanish once baselines receive the same tuning effort as the proposed method. An examination of neural recommender systems published in 2019 attempted to reproduce a set of recently published methods and reported that most could not be reproduced with reasonable effort, and that those which could were often matched or beaten by well-tuned simple baselines. A 2018 study of neural language models reported that carefully tuned standard LSTM architectures matched or exceeded the newer architectures that had been reported to supersede them. A study of generative adversarial network variants from the same period reported that, given a sufficiently large hyperparameter search, no variant consistently outperformed the original.

Three different subfields, one shape: the comparison was not between methods, it was between amounts of tuning. That is a claim about methodology rather than about honesty, and it is the single most useful prior to carry into reading a new method paper. It is also exactly what a real ablation is supposed to protect against.

The Chinchilla case discussed in that paper’s page is a different and encouraging variant: outside researchers found that one of three estimation methods could not be reconstructed from the published figures, published the reanalysis, and the headline conclusion survived because it had independent support. That is the system working, and it only worked because the numbers were published in the first place.

The frontier problem

For the largest models, none of the four levels is available to anyone outside the lab. The weights may be inaccessible, the training data undisclosed, the compute unaffordable and the method withheld. What is published is a set of evaluation results, and evaluation results are the part of a paper that is easiest to make look good without doing anything wrong.

This deserves to be said plainly rather than as an accusation: a large and commercially important portion of current AI progress is not externally verifiable, and readers should hold claims about it more loosely than claims about published open work — not because the labs are assumed to be lying, but because nothing forces the errors to surface. Reproduction is not a courtesy. It is the mechanism by which honest mistakes get caught, and where the mechanism is absent, mistakes stay.

Weighting a single result

Practical rules for what to do with an unreplicated finding you have just read.

• Weight by how much the claim needed to be true. A result that confirms what the authors set out to show carries less evidential weight than an incidental finding or a reported negative.
• Prefer results with an artefact. Released code and weights do not make a result correct, but they make it checkable, and checkable claims are the ones that get corrected.
• Discount margins that are within plausible noise. If the paper does not report variation across seeds, treat small improvements as unmeasured rather than as small.
• Look for the second, independent demonstration. One paper is a hypothesis. The thing that should move your beliefs is a different group reproducing the effect in a different setting, and that is worth waiting for on anything you would restructure a system around.
• Run it on your own task before believing it about your own task. Level four never transfers for free, and a small internal evaluation settles the question faster than any amount of further reading.