Context for RAG vs Context for Agents
4 min read · updated August 3, 2026
RAG and agents both spend their engineering effort on filling a window well. They are nonetheless opposite problems, and the property that separates them is whether the context survives the request.
Disposable versus accumulating
In a retrieval-augmented question-answering system, context is disposable. A question arrives, a retriever selects material, the model answers, and the assembled context is thrown away. The next question builds a fresh one from the same corpus. Nothing carries over, nothing accumulates, and the entire quality problem is selection: did the retriever find the right material and did the ordering let the model use it?
In an agent, context is accumulating state. Step 1 produces output that step 2 reads. A tool result at step 4 changes what step 9 should do. The context at step 20 is not a fresh selection from a corpus; it is the transcript of everything that happened, including the agent’s own actions and their consequences. The quality problem is not selection but maintenance: keeping the state accurate, current, non-contradictory and inside a budget while it grows.
That single difference — disposable versus accumulating — determines almost every downstream decision, which is why importing techniques from one to the other so reliably misfires.
The differences, in one place
| Dimension | Description |
|---|---|
| lifetime | RAG: one request. Agent: the whole run, sometimes across sessions. An agent needs a compaction and handoff story; a RAG pipeline needs neither. |
| dominant failure | RAG: the right material was not retrieved, or was retrieved and not used. Agent: the window filled, or it contains a superseded world-state the agent is acting on. |
| what fills it | RAG: retrieved documents, sized and ranked by you before the call. Agent: tool output, generated at run time by systems with no notion of your budget. |
| predictability | RAG: highly predictable — k chunks of roughly known size, so the budget is known before the request. Agent: unpredictable by construction; one tool call can consume the remaining window. |
| mutation | RAG: the corpus is read-only during the request. Agent: the agent edits the world it is reading, so earlier reads go stale mid-run. |
| ordering concern | RAG: position of retrieved documents relative to the question. Agent: recency and staleness of accumulated steps, plus keeping the current plan visible. |
| eviction | RAG: no eviction — the context is rebuilt each time. Agent: eviction is the central mechanism, and needs pins, scores and a stated priority order. |
| caching | RAG: only the static prefix caches; retrieved material changes every request. Agent: history is append-only, so a long stable prefix is available if nothing volatile is placed ahead of it. |
| cost shape | RAG: roughly constant per request. Agent: grows with the run, quadratically if history is re-sent unbounded. |
What happens when you mix them up
Both directions of confusion are common and each has a signature failure.
RAG techniques applied to an agent
The instinct is to treat the agent’s history as a corpus and retrieve over it: embed past steps, fetch the relevant ones, drop the rest. It sounds elegant and it breaks in a specific way — an agent needs recency and completeness of its recent state far more than it needs topical similarity. Retrieving semantically similar past steps will happily return step 3 while omitting step 19, and step 19 is the one that changed the file. Relevance is the wrong ranking function for state.
The related mistake is treating tool output like a retrieved document: scored, ranked, filtered by similarity. Tool output is not competing for relevance; it is the result of an action the agent took deliberately, and its value decays by step distance, not by similarity to the current question.
Agent techniques applied to RAG
The mirror mistake is accumulating context in a question-answering system: keeping the previous questions’ retrieved documents in the window in case they are useful again. This inflates every request, adds distractors from unrelated earlier questions, and produces the particularly confusing failure where an answer is drawn from a document retrieved for a different question two turns ago. If a conversation over a corpus needs history, keep the questions and answers, not the retrieved material behind them.
Building compaction and eviction machinery for a stateless RAG endpoint is the same error in a more expensive form: complexity serving a problem the workload does not have.
The hybrid, which is most real systems
Most production systems are both. An agent that can search documentation is doing retrieval inside an accumulating run, and the two contexts need to be managed separately even though they share a window.
- Retrieved material is disposable; treat it that way. Documents fetched at step 5 should not still be in the window at step 15. They were selected for one question, they answered it, and they should be evicted first. Retrieval results have the steepest justified decay of anything in an agent’s window.
- Give them separate allocations. One budget line for accumulating state and one for the disposable retrieval of this step, so a large retrieval cannot consume the room the plan needs — two blocks with independent floors in the allocator.
- Keep the conclusion, discard the evidence. When the agent has used a retrieved document, what should persist is what it concluded, not the document. “Confirmed from the API docs: rate limit is per-key, not per-IP” costs twenty tokens; the page it came from costs four thousand.
- Retrieve again rather than retain. Re-running a search at step 15 is usually cheaper than having carried the results through ten intervening steps, and it has the considerable advantage of returning current material.
The retrieval half of that — how the documents are found, chunked, ranked and cited — is the RAG cluster’s subject. What belongs here is only the part about the window they land in: how much room they get, how long they stay, and what replaces them when they go.
One diagnostic makes the distinction usable rather than theoretical. When something goes wrong, ask whether the request would have succeeded had it been the first request of a fresh session with the same inputs. If yes, the failure is accumulation and you are debugging an agent problem — superseded state, a filled window, an eviction that took the wrong thing. If no, the failure is selection and you are debugging a retrieval problem: the material was never found, or was found and placed badly. The two investigations share almost no steps, and starting the wrong one costs an afternoon. It is the first question in the diagnostic order for exactly that reason.
The same question is worth asking at design time, before anything has gone wrong. If your system’s hard cases are all “the model did not have the right document,” invest in retrieval and treat the window as a container. If they are all “the model had it three steps ago,” invest in state management and treat retrieval as a tool. Most teams can name which one describes their last five incidents, and that answer should be setting the roadmap.