Training Data Poisoning

The comfortable assumption about poisoning is that an attacker needs to control a meaningful percentage of a training corpus, which is implausible at web scale. Published work over the last three years has undermined that assumption in a specific and important way, and the numbers are worth knowing exactly.

Three different things called poisoning

Most discussion is about the first; most actual exposure is the third.

What the literature actually found

Two results changed the shape of this threat, and both are worth citing precisely rather than paraphrasing into a scare.

Poisoning web-scale datasets is practical. Carlini and colleagues published this in 2023, demonstrating two mechanisms against real, widely-used corpora. Split-view poisoning exploits the fact that large datasets are distributed as lists of URLs whose content can change after curation — buy an expired domain that a snapshot points at, and you control what downloaders receive. Frontrunning poisoning exploits predictable snapshot timing on crowd-edited sources: place content just before the crawl, revert after. Their estimate was that a small fraction of a percent of several standard datasets could have been poisoned for a modest sum.

The number of poisoned samples may be near-constant, not proportional. Anthropic, with the UK AI Security Institute and the Alan Turing Institute, published a study in October 2025 finding that a backdoor could be installed with roughly 250 malicious documents across models from 600M to 13B parameters — with the count not increasing as model and dataset size grew. If that behaviour holds more broadly, the intuition that scale dilutes an attacker is wrong in the direction that matters: a fixed, small effort suffices regardless of how large the corpus becomes.

Two caveats belong with that finding, and the authors state them. The backdoors studied were narrow behaviours rather than arbitrary capability, and the study covered a particular range of scales. It is a strong result about the feasibility of a class, not a demonstration that any behaviour can be installed at will.

Why web-scale corpora are reachable

The uncomfortable structural fact is that pretraining corpora are built from content anyone can write. Wikipedia, forums, code hosting, package registries, question-and-answer sites and personal blogs are all crawled, and all accept public contributions. An attacker does not breach anything; they publish.

They also do not need to know which corpus will pick it up. Content seeded broadly enough will be crawled by someone, and the same document may be ingested repeatedly across snapshots — which is why deduplication is a mitigation and near-duplicate variants are the countermeasure to it.

The version that affects you: retrieval poisoning

Almost nobody reading this trains a base model. Nearly everybody retrieves. Retrieval poisoning needs no training run and no patience: get a document into the index, and it enters contexts on demand.

The paths are mundane. A user-uploaded file that is indexed. A support ticket, a wiki page, a public web page your crawler picked up, a code comment in a repository the assistant reads. And retrieval poisoning is more targetable than training poisoning, because the attacker can write content optimised to be retrieved for a specific query — the semantic search is the delivery mechanism, and it is cooperative.

Treat index write access as a privileged operation. That single framing — the index is code, not content — produces most of the right controls on its own.

Persistent agent memory is the same problem wearing different clothes. A system that writes summaries of conversations back into a store it later reads has built a retrieval index whose write path is the untrusted conversation itself, with no review step at all. Anything written to memory should carry its provenance and should be treated, when read back, as content of the same trust level as whatever produced it — which for a session that touched a web page is untrusted.

Controls

If you fine-tune

• Curate from sources you control or trust, and keep the corpus reproducible with a manifest of hashes.
• Deduplicate aggressively, including near-duplicates, since repetition is how a small sample set gains influence.
• Review provenance for any dataset you did not assemble, and pin it by hash rather than by name.
• Evaluate for backdoors deliberately: standard benchmarks will not show one. Test on triggers you hypothesise, and compare behaviour against the base model on your own held-out set.
• Keep the base model and every training input inventoried, so a disclosed poisoned dataset can be traced to affected artefacts within an hour.

If you retrieve

• Authorise writes to the index. Anything user-submitted goes to a separate, lower-trust index — never mixed with curated documents.
• Partition by tenant at the index level, so a poisoned document cannot be retrieved into another customer’s context.
• Record provenance per chunk and surface it in the answer, so a strange response is traceable to a document rather than to the model.
• Apply the same untrusted-content posture to retrieved chunks as to any web page: they can carry instructions, and the context they enter should not hold the capabilities that make that interesting.
• Monitor for content that is retrieved unusually often or that appears for semantically unrelated queries — the signature of a document written to be retrieved.

For anyone consuming published weights rather than training them, the companion control is provenance on the artefact itself, covered in supply chain risk in open model weights.