Reproducibility of AI-Assisted Science: The New Failure Modes
5 min read · updated August 3, 2026
A method section is supposed to let a competent stranger repeat what you did. Machine learning inserts several hundred consequential choices between the data and the result, most of which are invisible in prose, and a handful of them can turn a null result into a striking one without anybody noticing.
Why this is a distinct problem
The general reproducibility difficulties in science — small samples, flexible analysis, publication bias — all apply here and are not the subject of this page. What is new is that the pipeline itself is now a large piece of software with a specific, recurring bug class, and that the bug produces results which look good rather than results which look broken. A leaked test set does not throw an error. It reports a high score, the score gets published, and the failure surfaces only when somebody tries to use the method and it does not work.
Published surveys have catalogued these failures across a wide range of scientific fields, and the durable contribution is the taxonomy rather than any count. What follows is that taxonomy, each entry paired with the thing to check. No claim is made here about how common any of them is, because that has not been established and inventing a percentage would be exactly the error the page is about.
The leakage taxonomy
| Failure | Description |
|---|---|
| No held-out set | Performance reported on data the model was fitted on. Check: does the paper name a test set that was untouched until the final evaluation? |
| Duplicate records | The same or near-identical sample appears on both sides of the split. Check: was deduplication done, and at what similarity threshold, and on what representation? |
| Preprocessing on all data | Normalisation, imputation, feature selection or dimensionality reduction fitted before splitting, so the training procedure saw the test set. Check: is every fitted transformation inside the cross-validation loop? |
| Proxy features | A feature encodes the label through the data collection process rather than through the phenomenon — the classic case being an imaging model that learned to recognise which site or which machine produced a scan, when site correlated with disease prevalence. Check: does performance survive a site-held-out evaluation? |
| Temporal leakage | Training data postdates the moment the prediction is supposed to be made. Check: for any task with a time axis, was the split chronological? |
| Group leakage | Multiple records from one patient, protein family, site or experiment are spread across the split. Check: what was the grouping unit, and was it respected? |
| Test-set model selection | Architecture, hyperparameters or a stopping point chosen by repeatedly evaluating on the test set. Check: was there a separate validation set, and how many configurations were tried? |
Splitting by the thing you want to generalise over
Most of the table collapses into one principle, and it is worth stating on its own because it turns a checklist into a decision you can make from first principles.
A random split estimates performance on new samples drawn from the same pool. That is almost never the question. The question is whether the method works on a new patient, a new hospital, a new time period, a new protein family, a new instrument. Whatever that noun is, the split has to be along it — group by patient, hold out an entire site, split by date, cluster by sequence identity. If you cannot say which noun your split is testing, the reported number does not correspond to any claim you would want to make.
The uncomfortable part is that doing this correctly usually lowers the number, sometimes dramatically. A large gap between the random-split score and the grouped-split score is not a disappointment; it is the measurement, and it is the single most informative thing a paper in this area can report.
One more thing belongs in the same section, because it is the check that catches whatever the split design missed: a baseline. Fit the simplest defensible model — a logistic regression on a handful of obvious features, or a rule a domain expert would write down — on the identical split, and report it next to the result. This costs an afternoon and does two jobs at once. It tells a reader how much of the performance is attributable to the method rather than to the problem being easy, and it is a leakage detector: a trivial baseline scoring implausibly well is the clearest available sign that something in the pipeline is leaking the label. A surprising number of reported gaps in applied machine learning narrow considerably against a well-tuned simple model, and finding that out before publication is much cheaper than after.
The moving-target problem
A newer difficulty applies to any study whose method involves calling a hosted model. Three things can move underneath you.
- The weights. A model name is a pointer, and what it points at can be updated. Two runs months apart can produce different results with the same code, the same prompt and the same name in the method section.
- The availability. Hosted models are retired. A study whose central method is a call to a model that no longer exists cannot be repeated at all, only re-run against a successor, which is a different experiment.
- The determinism. Setting temperature to zero reduces variation but does not guarantee identical output across runs or infrastructure, for reasons covered in why temperature zero is not deterministic. A method that assumes it is will report results nobody can match exactly.
The mitigations are ordinary but have to be deliberate: pin the most specific version identifier the provider exposes rather than an alias; record the identifier that actually served each call, not the one you requested; publish prompts, parameters and seeds verbatim rather than describing them; report the number of repeats and the variation across them; and where the finding matters, replicate it on an open-weights model that can be archived, which is the argument developed in open science and open models.
What a method section has to contain
- The dataset, its version, and the exact inclusion and exclusion criteria — with counts at each filtering step.
- The split: what unit, what proportion, chosen how, and whether the test set was looked at more than once.
- Every transformation that is fitted to data, and whether it was fitted inside the split.
- The full hyperparameter search space and the number of configurations evaluated, not only the winning configuration.
- The comparator: what a simple baseline achieves on the identical split. A result with no baseline is uninterpretable, and a surprisingly large number of published gaps disappear against a well-tuned simple model.
- For any hosted model: the exact version, the date, the prompts, the sampling parameters, and the number of repeats.