Context Poisoning and Bad Retrieved Data

Grounding a model in retrieved documents is the most reliable hallucination fix available. It also hands anyone who can write to your corpus a direct channel into your model’s answers. These are the same property viewed from two sides, and you cannot have the first without accepting the second.

Context wins, by design

When retrieved text contradicts what a model absorbed in training, the retrieved text usually wins. This is not a defect — it is the entire reason retrieval works. In-context information is specific, recent and local to the request; parametric knowledge is a compressed average over a corpus with a cutoff. Every instruction-tuned model has been trained to weight the former, because a model that argued with its own context would be useless for summarisation, for document QA and for tool use.

The consequence: your corpus is not a hint. It is closer to an override. Anything that reaches the context window with the shape of an authoritative statement is likely to come back out as one, and the model has no mechanism for auditing provenance because provenance is not a feature of text.

The published attack

Zou, Geng, Wang and Jia’s PoisonedRAG (2024) is the reference result. The threat model is realistic: the attacker cannot touch the model or the retriever, only inject a small number of texts into the knowledge corpus — the situation you are in the moment you index a public wiki, a customer-editable knowledge base, scraped documentation or user-submitted content.

The construction is a two-part optimisation. Each poisoned passage must be retrievable for the target question, which is a similarity objective, and must induce the chosen answer once retrieved, which is a generation objective. The paper reported that injecting on the order of five crafted passages per target question into a corpus of millions achieved a high targeted attack success rate — around 90% in their experiments — across the retrievers and models they tested.

The number that matters is the ratio. Five documents in millions is a vanishing fraction of the corpus, which means corpus-level integrity statistics will never see it. The attack is targeted: it changes the answer to one question and leaves everything else alone. Aggregate quality monitoring is structurally blind to it.

Carlini et al.’s Poisoning Web-Scale Training Datasets Is Practical (2023) makes the parallel point one level up, for pretraining corpora. Same lesson, different budget: whoever can write to the data can influence the output.

Bad retrieval without an attacker

Most context poisoning is accidental, and two published results are more useful here than any amount of intuition.

Shi et al.’s Large Language Models Can Be Easily Distracted by Irrelevant Context (2023) added a single irrelevant sentence to arithmetic word problems and reported substantial accuracy drops. The irrelevant material did not have to be misleading — merely present.

Cuconasu et al.’s The Power of Noise (SIGIR 2024) produced the counterintuitive companion finding: in a RAG pipeline, documents that are related but not relevant — the near misses a good retriever surfaces at ranks 3 to 10 — hurt accuracy more than documents that are entirely random. Random noise is easy to ignore; a plausible neighbour is not. This is an uncomfortable result for the standard instinct of raising top_k to improve recall, because the marginal document you add is precisely a near miss.

Add to these the position effects from long-context degradation — material in the middle of a long context is used less reliably than material at either end — and the picture is that context is a scarce, ordered resource and stuffing it is not free.

When the data contains instructions

The sharper version of the problem: retrieved text that is written as a directive. Greshake et al.’s indirect prompt injection work (2023) established the shape — a payload sits in a document, a web page or an email, waits to be retrieved, and issues instructions that the model has no reliable way to distinguish from yours. Every token in the context window is, to the model, the same kind of thing.

The mitigation is architectural rather than linguistic. Keep retrieved content in a clearly delimited region, tell the model in the system prompt that everything in that region is data to be summarised or cited and never obeyed, and — most importantly — do not grant the response the authority to trigger side effects on its own. If a tool call can be emitted as a consequence of retrieved text, the retrieved text has your permissions. Put a confirmation or a policy check between the model and anything irreversible.

Defences that survive contact

• Provenance on every chunk, with a trust tier. Curated documentation and a scraped forum thread should not enter the context as peers. Rank by trust as well as by similarity, and put the tier in the context so the model can weight it and your logs can explain an answer.
• Review the write path, not just the read path. Who can add a document? Is there a diff? An LLM corpus with an unreviewed write path is a production dependency with an unreviewed write path.
• Retrieve fewer, better chunks. Given the near-miss result, a reranker that cuts ten candidates to three usually beats raising the limit. Measure it on your own set rather than assuming.
• Verify claims against the span, always. The entailment gate from the grounding page catches nothing about a poisoned document — the claim really is entailed by the passage — but it catches everything about a model that has blended a poisoned passage with something else, which is the more common failure.
• Keep a canary set. A few dozen questions with known answers, run against the live corpus on a schedule. A targeted poisoning that moves one answer is invisible to aggregate metrics and obvious to a canary that happens to cover it — so choose canaries for the questions where a wrong answer would cost the most.
• Ablate when quality drops. When answers degrade after an ingest, re-run the evaluation with the new source excluded. Per-source ablation localises corpus regressions faster than any amount of prompt work.