Benchmark Contamination: When Models Have Seen the Test

A benchmark measures capability only if the model has not memorised the answers. Since benchmarks live on the public web and training corpora are scraped from the public web, the default assumption for any test published before a model’s cutoff should be that some of it leaked.

How contamination happens

Rarely by anyone cheating. The usual routes are mundane:

• The dataset itself was crawled. Benchmarks live in GitHub repositories and on Hugging Face. Both are crawled.
• Discussion carries the answers. Blog posts, papers, Stack Overflow threads and tutorial notebooks quote benchmark items with their solutions, so an item can leak even if the canonical file was excluded.
• Derived data. Synthetic training sets generated by another model, or instruction-tuning mixes assembled from public corpora, routinely contain paraphrases of benchmark items. Exact string deduplication does not catch these.
• Evaluation-driven iteration. Even with a clean training set, repeatedly tuning against a public test set overfits it through the experimenter. This is contamination of the process rather than of the data, and it is invisible to every string-match check.

Why it is worse than a wrong number

If contamination inflated everyone equally, rankings would survive and only the absolute level would be wrong. It does not. Each model has a different corpus with a different cutoff, so contamination is a different additive term for every row of the leaderboard — which scrambles the ordering rather than shifting it. That is precisely the quantity you were trying to read.

It also decays in a specific direction over time: an old benchmark gets more contaminated with every new model generation, so a benchmark’s usefulness has a half-life measured in model releases. This is the argument behind continuously-refreshed benchmarks such as LiveCodeBench, which tags problems by their publication date so you can score only on problems released after a given model’s cutoff, and LiveBench, which replaces questions on a rolling schedule.

Published detection methods

Two things to know before reaching for any of them. n-gram overlap, the method the large lab reports historically used, is a lower bound and a weak one: it needs corpus access you almost never have, and a benchmark item that was reworded anywhere on the web passes it untouched. And Min-K% Prob and its relatives are membership-inference methods, which means they answer “was this exact text seen” rather than “is this score inflated”. A model can be contaminated by having seen a discussion of a benchmark rather than the file, and no membership test on the file will find it.

Two tests that work through an API

Most detection literature assumes corpus or log-probability access. Two families do not, and they are the ones worth knowing if you are evaluating a closed model.

Completion probing

Take a benchmark item, cut it at a point where completion requires having seen this exact item rather than merely understanding the topic, and ask for the rest. The signal is not a correct answer — it is verbatim reproduction of incidental detail: the same distractor options in the same order, the same variable names, the same odd phrasing.

PROBE = """You are completing an item from a public evaluation dataset.
Dataset: {dataset}    Split: {split}
Here is the first part of one instance:

{prefix}

Reproduce the remainder of this instance exactly as it appears in the
dataset, including any answer options and their original order."""

# Score: character-level similarity between the completion and the true
# remainder, compared against the same score for items you KNOW postdate
# the model's cutoff. The absolute number is meaningless; the gap is not.

The control group is the whole method. Run the same probe on items created after the model’s stated cutoff to establish what similarity looks like when memorisation is impossible, then look at whether the pre-cutoff items sit above that band.

The temporal split

Wherever a benchmark carries dates — GitHub issues, competition problems, exam papers, news — split it at the model’s cutoff and score the halves separately. A capability gap between two halves drawn from the same distribution, differing only in whether the model could have seen them, is the cleanest contamination evidence available without corpus access. This is the design principle behind LiveCodeBench, and you can apply it to any dated corpus you have.

Two cautions. Difficulty is not always stationary in time — a competition may simply have got harder — so check the human success rate on both halves if you have it. And a model with retrieval or web access breaks the test entirely; run these probes against the bare model.

Neither test yields a clean verdict, and it is worth being clear about what you get. The output is evidence about whether a particular benchmark score is trustworthy for a particular model, not a certificate either way, and the sensible action on a positive result is almost never to accuse anyone. It is to stop using that benchmark for that model and move the decision onto something dated after the cutoff — which is the same action you would take if the benchmark had simply saturated, and is why contamination is best treated as one more reason a public number has expired rather than as a scandal.

Protecting your own eval set

Once your private eval matters, it becomes contaminable too, and the leak paths are ordinary engineering ones.

• Never commit it to a public repository, never paste it into a public issue, and keep it out of any documentation site you publish. A private eval that appears in a crawl is a public benchmark with worse coverage.
• Check the retention terms of every endpoint you send it to. Some providers train on API traffic by default, some do not, some vary by plan. This is a question with a documented answer per provider and it is worth answering before your held-out set has been posted to nine of them.
• Keep a sealed slice. Hold back 20% that is used only for final decisions, not for iteration. The experimenter-overfitting route is real, and a slice you have never optimised against is the only defence against it.
• Refresh from production. Items collected from last month’s incidents cannot have been in any training corpus. This is the strongest structural argument for building your eval from a continuously curated golden set rather than freezing one once.