Retrieval-Augmented Generation: The Original Paper

“RAG” now describes almost any system that puts retrieved text into a prompt. The paper that introduced the term described something considerably more specific, and the difference is not pedantry — it explains why several things people expect from RAG do not happen.

What the paper proposed

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) addresses a limitation of pre-trained sequence models: their knowledge is baked into their parameters, cannot be inspected, cannot be updated without retraining, and cannot be cited. The proposal is a hybrid — a parametric memory, meaning the model weights, plus a non-parametric memory, meaning a dense vector index over a document collection.

Concretely, the architecture combines a dense passage retriever with a sequence-to-sequence generator. Given an input, the retriever encodes it into a query vector, finds the top-k passages from a dense index built over a Wikipedia dump, and the generator produces the output conditioned on the input together with those passages. The retrieved documents are treated as a latent variable, and that framing is what makes the rest of the paper work.

Sequence and token variants

Because the retrieved passage is latent, the model has to marginalise over the possibilities rather than commit to one, and the paper defines two ways to do it.

Notice that this is a genuine architectural distinction with an observable consequence, and that it has no analogue in a stuff-the-prompt pipeline. When you place five passages in a context window and ask a model to answer, there is no marginalisation happening anywhere: the model attends over everything at once and you have no handle on which passage produced which claim. That is a different system, not a simplified version of this one.

The part everyone dropped

Here is the core of it. In the paper, the retriever and the generator are fine-tuned jointly, end to end, on the downstream task. The query encoder and the generator receive gradients from the final objective. The document encoder and its index are kept fixed, for the practical reason that re-encoding and re-indexing the whole corpus after every update would be prohibitive, and the paper says so.

So the retriever learns what “relevant” means for the task it is serving. That is a fundamentally different object from an off-the-shelf embedding model chosen because it topped a leaderboard. The paper reported strong results on open-domain question answering against extractive systems of the time, and the joint training is a material part of why.

Almost no production system called RAG does this. Retrieval is a separate service with a separately chosen embedding model, and nothing about the generation objective ever reaches the retriever. That is a defensible engineering choice — it is modular, debuggable, and does not require a training run — but it means the retriever is optimised for a proxy, and a great deal of the practical difficulty in evaluating a RAG system comes from that gap.

How the term drifted

The mechanism of the drift is worth naming because it recurs. A paper introduces a name for a specific architecture. The architecture’s most legible feature — “it looks things up before answering” — is much easier to reproduce than its actual method. Instruction-tuned models arrive with context windows large enough that looking things up and pasting them in works acceptably well with no training at all. The name attaches to the easy thing, and within a couple of years the term denotes the pipeline pattern rather than the model.

This is not a scandal and nobody misbehaved. But it produces a specific confusion: people read the original results, or summaries of them, and expect prompt-stuffing to inherit properties it does not have. It is the same failure mode as citing an encoder-decoder translation paper for the architecture of a modern chat model — the ancestor is real, the identity is not.

Why the distinction still matters

• Attribution is not free. In the paper, the retrieved document is part of the generative model’s probability, so provenance is structural. In a prompt pipeline, a citation is something you ask the model to produce and then have to verify, because it can and will cite a passage it did not use. That is the whole subject of citation faithfulness.
• Retriever quality is your problem. With no joint training, nothing aligns the retriever to the task, so relevance tuning becomes manual work: chunking, hybrid search, reranking, metadata filters. Every one of those is compensating for the gradient the original architecture had and yours does not.
• “RAG is dead” arguments target the pipeline. Claims that long context makes retrieval obsolete are about prompt-stuffing. They say nothing about learned retrieval, and they also collide with the cost arithmetic of re-sending a large corpus on every request.

The general reading skill here is simple and widely applicable: when a term is used loosely, find the paper that coined it and check what it actually named. Half of the arguments in a field are two people using one word for two systems.