Fine-Tuning vs RAG: They Solve Different Problems

The comparison is usually framed as a cost or accuracy trade-off. It is not. Retrieval and fine-tuning change different things, and the useful question is which thing is broken.

The one distinction that matters

Retrieval changes the context. At inference time, something outside the model finds relevant text and puts it in the prompt. The weights are untouched, the facts are current as of the last index refresh, and you can point at exactly which document produced the answer.

Fine-tuning changes the weights. It shifts the distribution the model samples from — towards your formats, your register, your task decomposition, your label vocabulary. It does not attach a source, it cannot be updated without another training run, and it is applied uniformly to every request.

So: if the model is wrong because it lacks information, retrieval. If it is wrong because it does the right thing badly, fine-tuning. Almost every bad outcome in this area is a team applying one to the other problem.

There is an operational corollary that decides more real architectures than the quality argument does: the two have completely different update latencies. Correcting a fact in a retrieval system means editing a document and reindexing, which is minutes and can be done by someone who is not an engineer. Correcting a fact in a fine-tuned model means assembling a corrected dataset, retraining, re-evaluating all four suites and redeploying, which is weeks and requires the team that did it the first time. If the thing you need to correct changes at all, that gap is the whole decision.

Mistake one: training to install facts

The reasoning goes: our documentation is 40,000 pages, we will fine-tune the model on it, and then it will know our product. What actually happens is that the model learns the style of your documentation — the section headings, the cadence, the hedging — and produces fluent, well-formatted, confidently wrong answers about it.

The mechanism is not mysterious. A fact appearing once in a corpus of 40,000 pages contributes a vanishing share of the gradient signal. Stylistic regularities appear on every page and contribute to every batch. Gradient descent is a frequency machine; it learns the frequent thing.

The tell is distinctive and worth learning to spot: after a knowledge-motivated fine-tune, answers get better looking and no more accurate, and the model becomes noticeably more willing to answer questions it should refuse. That second half matters more than the first.

Mistake two: retrieving to fix behaviour

The mirror image is subtler. The model produces correct content in the wrong shape — prose where you wanted JSON, four paragraphs where you wanted one line, a British register where the brand is American. The instinct is to retrieve more: add style-guide passages to the context, add exemplars to the context, add the schema to the context.

This works, in the sense that it moves the metric, and it accumulates into a system prompt that is 3,000 tokens of instruction paid on every single request. It is also fragile: instructions buried in a long prompt are followed less reliably than instructions the weights encode, and every new edge case adds another sentence that competes with the ones already there.

Format is the thing fine-tuning is genuinely good at. If your prompt has grown a style section longer than the actual task description, that section is a fine-tuning dataset that has not been written down yet.

What the published comparisons found

Two papers are worth reading in full here rather than being summarised into a bullet.

• Ovadia et al., 2023 — Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (arXiv 2312.05934). Compared unsupervised fine-tuning against retrieval for injecting new factual knowledge, and reported retrieval ahead on the knowledge-intensive tasks tested — including on facts the model had seen during pretraining.
• Gekhman et al., 2024 — Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? (arXiv 2405.05904). Found that examples introducing genuinely new knowledge are fitted slowly, and that as the model does fit them its tendency to hallucinate on other questions increases. This is the best available explanation for the “confidently wrong, and now more willing to guess” symptom above.

Neither paper says fine-tuning is useless. Both say it is the wrong instrument for knowledge, which is exactly the split this page is about.

They compose, and usually should

The mature configuration is both, doing their own jobs. Retrieval supplies the facts, and a fine-tune teaches the model how to use retrieved context: when to cite, how to say “the provided documents do not answer this”, how to reconcile two passages that disagree, what to emit when retrieval returns nothing.

That last behaviour is a real fine-tuning target and a good one, because abstention is a behaviour and not a fact. Training a model to decline when the context is insufficient is teaching it a policy, which is precisely what weight updates encode well.

The dataset for that fine-tune has a specific shape, and it is worth spelling out because it is not obvious. Each example is a full retrieval-augmented prompt — the same system prompt, the same retrieved passages in the same order, the same formatting your production pipeline produces — paired with the response you wanted. Crucially it must include the negative cases: prompts where retrieval returned nothing useful, or returned contradictory passages, with a target that declines or flags the conflict. A dataset made only of successful retrievals teaches the model that retrieved context is always sufficient, which is the opposite of the lesson.

Picking, in four questions