Pattern: Cache the Expensive, Recompute the Cheap

Caching model output is unusually attractive because the calls are unusually expensive, and unusually dangerous because the same input can legitimately deserve a different answer tomorrow. Both facts belong in the same decision, and most implementations only weigh the first.

The decision rule

For a given call site, cache when the expected saving exceeds the expected cost of serving something stale. Written out, with your own numbers:

  h  hit rate                      fraction of calls that find a valid entry
  C  cost of the call               money, and latency if a user is waiting
  s  staleness rate                 fraction of hits whose stored answer is no
                                    longer the answer that would be produced
  L  cost of one stale answer       from "nobody notices" to "wrong price shown"
  k  cost of the cache itself       storage, lookup, and the engineering to
                                    keep invalidation correct

  cache when:    h * C   >   h * s * L  +  k
       i.e.      C       >   s * L  +  k/h

Two readings:
  · h scales the benefit but NOT the staleness penalty per hit — a low hit
    rate makes the cache pointless without making it safe.
  · s * L is the term that decides most real cases. If a stale answer is
    harmless, cache aggressively. If a stale answer is a wrong number in
    front of a customer, C has to be very large indeed to justify it.

The rule’s value is that it forces two estimates that are usually skipped. h is measurable before you build anything — hash your last week of inputs and count repeats, which takes an hour and frequently ends the discussion in either direction. And s is a question about your domain that somebody knows the answer to, provided they are asked.

Two refinements are worth applying before you accept the answer. The first is that C should include latency where a user is waiting, not only money: for an interactive feature the value of removing a multi-second wait frequently exceeds the token cost, and a cache that looks marginal on price alone can be clearly worth it on that basis. The second is that h is not a constant. It is a property of your traffic mix, and it moves when you launch in a new market, change the interface that generates the inputs, or acquire a customer whose usage pattern differs from everyone else’s. Measuring it once at design time and never again is how a cache stays in the codebase long after it stopped paying for itself.

The term everyone omits

Staleness is not a property of the cache. It is a property of the relationship between the input and the answer, and it varies enormously between call sites that look identical in code.

The fourth row is the one that causes production surprises. A team improves a prompt, deploys it, and quality does not change, because most traffic is being served from a cache populated by the old prompt. The bug is invisible in every dashboard: cost is down, latency is good, and the improvement simply did not happen.

The key is the correctness problem

Nearly every incorrect cache is a key that omitted something the answer depended on. The discipline is to enumerate every input to the answer and include all of them, then decide deliberately what to leave out.

key = hash(
  normalised_input,      // trim, collapse whitespace, case-fold IF the answer
                         //   genuinely does not depend on case
  prompt_id + version,   // omit this and a prompt fix does not take effect
  model_id + version,    // omit this and an upgrade is silently rolled back
  params,                // temperature, max_tokens, schema version
  tenant_or_scope,       // omit this and one customer sees another's answer
  locale,                // omit this and the cache decides everybody's language
  data_version,          // for retrieval: the index or document revision
)

Two of those lines are security properties rather than correctness niceties. A cache key without a tenant scope is a cross-tenant data leak with a plausible-looking cause, and it is the most damaging bug available in this pattern. If entries are shared across users by design — a public FAQ answer, a translation — that sharing should be an explicit decision recorded at the call site, not a consequence of which fields somebody remembered.

Normalisation deserves a moment of thought rather than a reflex. Aggressive normalisation raises the hit rate and quietly merges inputs that deserved different answers; conservative normalisation is correct and may leave the hit rate too low to matter. Decide it per call site based on whether the discarded distinction can change the answer.

Invalidation, by what changed

Time-based expiry is the default and it is the weakest option, because a TTL is a guess about staleness rather than knowledge of it. Prefer, in order:

• Content-addressed, never invalidated. If the key contains a hash of everything the answer depends on, a change produces a different key and the old entry is simply unused. No invalidation logic exists to be wrong. Reach for this first; it is available more often than people expect.
• Dependency-based. Record which entities an entry depends on and drop it when one changes. More work, and it is the correct answer for anything derived from your own data, because you already have the write path where the invalidation belongs.
• Version-prefixed. Put a global version in the key and bump it to drop everything at once. The blunt instrument, and genuinely useful for a prompt or model change where the honest scope of invalidation is “all of it”.
• Time-based. Last resort, for the case where the answer depends on the world and you cannot know when the world changed. Choose the TTL from how long a wrong answer is tolerable, not from how long you would like the saving to last.

There is one further decision that belongs with invalidation rather than after it: what happens on a miss when the entry has just been dropped. If a popular entry expires and a hundred requests arrive before the recomputation finishes, all hundred will call the model unless something stops them. A single-flight lock — the first caller computes, the rest wait for its result — turns that into one call, and it matters far more here than in an ordinary cache because each of the hundred duplicate calls is a purchase rather than a wasted database read.

Serving stale content while refreshing in the background is worth knowing about here too, because it decouples the latency benefit from the freshness cost. The cache-layer mechanics — where each layer sits, what it keys on, how they compose — are treated separately; this page is only about the decision to cache at all.

What not to cache

• Anything where variety is the point. Creative suggestions, brainstorming, alternative phrasings. A user who asks twice and receives the identical answer both times concludes the feature is broken, and they are not wrong.
• Long-tail inputs. Free-form questions rarely repeat exactly. Measure h before building; a cache with a very low hit rate is pure k with none of the benefit, and semantic matching to raise the rate imports a new correctness risk — two questions that embed similarly are not necessarily two questions with the same answer.
• Anything personalised that is not scoped. If you cannot enumerate the personalisation inputs confidently enough to put them in the key, do not cache. An incomplete key here is the leak described above.
• What the provider already caches. A long stable prefix may be discounted upstream, which is a different mechanism with different economics — prompt caching reduces the cost of a call you still make, whereas this pattern is about not making it. They compose, and confusing them leads to building the harder one when the easier one was available.