Agentic RAG: Letting the Model Decide When to Search

In a classical RAG pipeline, retrieval is unconditional: every request searches, whether or not searching helps. Handing the model a search tool turns that into a decision — and decisions have both a cost curve and a failure mode.

Two pipeline shapes

Always-retrieve is a fixed graph. Query in, search, concatenate, generate, answer out. One generation call, predictable latency, predictable cost, and the retrieval either helped or wasted some tokens.

Decide-then-retrieve gives the model a search(query) tool and lets it choose. It can skip retrieval for “rewrite this paragraph”, search twice with different terms when the first attempt returns nothing useful, and decompose a multi-hop question into sequential lookups where the second query depends on the first result — which is the one thing a fixed pipeline genuinely cannot do.

The research line here is worth knowing. Self-RAG (Asai et al., arXiv:2310.11511) trains a model to emit reflection tokens deciding whether to retrieve and whether the retrieved passage is relevant. FLARE (Jiang et al., arXiv:2305.06983) triggers retrieval mid- generation when the model’s own confidence in the next sentence drops. Both are more disciplined than “give it a tool”, and both make the same bet: that the decision to retrieve carries information.

The cost model

Write it out with symbols and substitute your own numbers. Let r be the fraction of requests that genuinely need retrieval, Tc the tokens a retrieved context adds, Td the tokens the tool-definition and decision turn adds, and Gin the input price per token.

always     = Tc * Gin
agentic    = Td * Gin  +  r * (Tc * Gin)  +  r * (extra turn overhead)

agentic wins when:   Td  <  (1 - r) * Tc  -  r * overhead

Assume Tc = 3,000 (six chunks), Td = 250 (tool schema + decision):

  r = 0.9  ->  saving 0.1 * 3000 = 300 tokens, cost 250. Break-even.
  r = 0.5  ->  saving 1500, cost 250. Clearly worth it.
  r = 0.2  ->  saving 2400, cost 250. Strongly worth it.

The entire question is r, and r is knowable: sample two hundred real requests and label whether the corpus was needed. A documentation assistant where every question is about the docs has r near 0.95 and should not bother. A general assistant that happens to have a knowledge base attached might sit at 0.2, where four out of five requests are currently paying for context they ignore.

Note the second-order effect that does not appear in the arithmetic: unnecessary context is not merely wasted, it is actively harmful. Every irrelevant chunk is a distractor, and the failure taxonomy on the generation page is full of things that only happen because something plausible and wrong was in the context. Skipping retrieval when it is not needed is a quality improvement as well as a saving.

Latency runs the other way

The cost model favours agentic retrieval at low r. Latency does not. Always-retrieve is one round trip to the model, with the search overlapping nothing. Decide-then-retrieve is at minimum two — a decision call, then the answer call — and every additional search the model chooses adds another, sequentially, because it cannot issue the second query until it has read the first result.

So the shape of the trade is: agentic retrieval reduces cost and improves quality on mixed workloads while increasing p95 latency, and the increase is unbounded above unless you bound it. If your product is a chat interface where the user is reading a streamed answer, the first token now arrives after two or three round trips instead of one, and that is felt.

A common compromise: run retrieval speculatively in parallel with the decision call, and discard the result if the model says it does not need it. You pay the retrieval compute either way but not the context tokens, and the round trip is hidden. Retrieval compute is cheap; context tokens are not.

What the loop breaks

• Confident non-retrieval. The model decides it knows the answer and does not search. This is the defining failure of the pattern, it is silent, and it is worst on exactly the questions where its training data contains a plausible generic answer to your proprietary question. Always-retrieve has no equivalent failure.
• The query it writes is worse than the user’s. Tool-calling models tend to write short, keyword-ish search arguments. Against a dense retriever that expects natural language, this measurably changes what comes back — and it happens invisibly, because the tool argument is rarely logged.
• The search-again spiral. Nothing useful comes back, so the model rephrases and searches again, five times, each attempt adding its failed results to the context. Cost grows, context fills with noise, and the eventual answer is worse than if it had stopped after one.
• Cost variance. Your per-request cost is now a distribution with a long tail rather than a number, which makes capacity planning and per-customer pricing harder. Track p95 request cost, not the mean.

Guardrails

• Hard cap the iterations. Three searches, then answer with what you have. Enforce it in the loop, not in the prompt.
• Log the tool arguments. The query the model wrote is the most diagnostic single field in an agentic RAG system and the one most often missing from traces.
• Deduplicate across iterations. Repeated searches return overlapping chunks; without deduplication the context fills with the same text three times over.
• Describe the tool by corpus, not by capability. “Searches the internal engineering handbook, which covers deploys, on-call and incident process” produces far better decisions than “searches the knowledge base”, because the model is deciding on the basis of the description alone.
• Make retrieval mandatory for known categories. If a question mentions a product name, pricing or policy, retrieve regardless of what the model wanted. A hybrid of the two shapes beats a purist version of either.

Evaluate the decision separately from the answer. Label a couple of hundred real requests with whether retrieval was needed, then score the model’s choice as an ordinary binary classifier and look at the two error types independently. They are not symmetric: a false positive costs you 3,000 tokens and some distraction, while a false negative costs you a confidently wrong answer with no indication that anything was missed. If the confusion matrix shows false negatives at any appreciable rate, bias the tool description toward retrieving and accept the wasted tokens — the errors are not worth the same and the threshold should reflect that.