What to Put in the System Prompt vs Retrieve on Demand

“Keep the system prompt short” is not a rule, it is a folk memory from when windows were small. The real question is arithmetic: a block used on most requests may be cheaper to carry always than to fetch sometimes, and prompt caching moves the threshold a long way.

The question, stated properly

You have a body of reference material — a product catalogue, a policy document, a set of API definitions, a style guide. It is too large to be trivial and too small to be obviously impossible. Two placements are available:

• Static. Put it in the system block, sent on every request. Always present, zero retrieval latency, zero chance of retrieving the wrong part, and it sits at the front of the prompt where a cache can hold it.
• Retrieved. Fetch the relevant fragment per request. Smaller payload most of the time, but a retrieval step that costs latency and can miss.

The instinct is that retrieval is obviously better for anything large. That instinct is from a world without cached tokens. Write the two cost structures down and the answer moves.

Two cost structures

Let S be the size of the block in tokens, p the fraction of requests that actually need any of it, f the fraction of S that a retrieval returns when it fires, P the uncached input price per token, and r the cache read multiplier — the fraction of P that a cached token costs. All of these are your inputs; nothing here quotes a provider.

static, cached prefix    : cost = S * P * r          (every request)
static, uncached         : cost = S * P              (every request)
retrieved                : cost = p * (f * S) * P + R
                                  where R = embedding/search cost per request

The mechanics of the cache discount belong to cached tokens; here it is just the parameter r. What matters is that r is substantially less than 1, which means the static option is not charged at face value — and that changes the comparison from “always pay S” to “always pay a fraction of S.”

The break-even in hit rate

Ignore R for a moment and set the two expressions equal. Static-cached beats retrieval when:

S * P * r  <  p * f * S * P
            r  <  p * f
            p  >  r / f

The block size cancels. That is the surprise in this derivation and it is worth sitting with: how big the block is does not enter the decision at all — only how often it is needed, how much of it a retrieval pulls, and what a cached token costs relative to an uncached one. Size only matters through the window, not through the bill.

Put numbers in as illustration, marked as assumptions. If a cached read costs a tenth of an uncached one (r = 0.1) and a retrieval returns a quarter of the material when it fires (f = 0.25), then static wins whenever p > 0.4 — that is, whenever more than about 40% of requests touch the material at all. If your cache discount is weaker, say r = 0.25, the threshold moves to p > 1.0, which is to say static never wins on cost alone and retrieval is correct.

Two adjustments make it realistic. First, R is not zero: an embedding call plus a vector search per request has a real cost, and adding it shifts the threshold down in favour of static. Second, a cache only helps if it is being hit, which requires the traffic to be frequent enough to stay inside the cache’s time-to-live and the prefix to be genuinely stable — prefix stability is a design constraint of its own. Low-volume applications should evaluate with r = 1, which almost always selects retrieval.

What the arithmetic does not cover

Cost is one of three axes, and it is often not the deciding one.

The rule

Combining the derivation with the three axes gives a placement rule you can apply without recomputing every time:

• Static, always: anything whose absence changes behaviour rather than quality — role, safety policy, output contract, refusal rules, tool semantics. These are usually small and the argument is not economic.
• Static, if p > r/f and it is stable: reference material used on most requests. Put it at the very front, ahead of anything that varies, so it forms a cacheable prefix.
• Retrieved: anything with a long tail of rarely needed detail, anything that changes faster than your cache time-to-live, and anything so large that carrying it would crowd out the history and tool output the session needs.
• Neither: material that fails both tests is often material that should be a tool the model calls rather than text it reads — a lookup function beats both placements for anything with a precise key.

Two refinements are worth applying once the rule is in place. The first is that p is measurable rather than guessable: log, for a week, whether each request’s answer actually depended on the material, or approximate it with whether a retrieval over the same corpus would have returned anything above threshold. Applications routinely discover that a block they were carrying on every request is needed on eight percent of them, or that one they were retrieving is needed on ninety.

The second is that the decision is per block, not per corpus. A policy document is rarely one thing: a two-page summary of the rules that apply to every request plus forty pages of edge cases that apply to almost none. Splitting it puts the summary in the static prefix, where it is cheap and always present, and leaves the edge cases behind a retrieval step where their cost is paid only when they matter. That split is usually available and usually better than either pure placement, and it is the reason the rule is stated in terms of blocks rather than sources.