Pattern: Precompute Overnight, Serve Instantly

A model call at request time costs latency the user feels and money you spend on demand. The same call run overnight costs the same money at a possibly better rate and zero latency, in exchange for one concession: you have to know what to compute before anyone asks.

What you are actually trading

Precomputation looks like a caching variant and is a different trade. A cache computes on the first request and reuses; precomputation computes before any request, which means paying for answers nobody may ever read. That sounds strictly worse and frequently is not, for four reasons.

• Latency goes to a database read. Not faster inference — no inference. This is the only technique in the cluster that removes the model from the request path entirely, and it is what makes an AI-derived field feel like an ordinary product feature.
• Batch work can use cheaper capacity. Where a provider offers a discounted asynchronous tier, work with no deadline qualifies — the discount is the whole point of that tier and precomputed work is its natural customer.
• Failures stop being user-visible. A failed batch item is retried tomorrow. A failed request-time call is an error in front of somebody. Moving the call moves the entire reliability problem into a place where retries are cheap and nobody is waiting.
• Spend becomes predictable. Batch volume is known in advance, so the bill is a number you set rather than a number your traffic sets. For a feature whose unit economics are marginal, converting a variable cost into a planned one can be worth more than the saving itself.

The eligibility test

Four questions. A no to any of them means this feature cannot be precomputed as it stands — though sometimes the right response is to change the feature.

The second question deserves a note, because “stable” is usually answered from intuition and is checkable. Take the entities you would precompute over and measure the distribution of time between writes. In most catalogues that distribution is heavily skewed: the large majority of records are never touched after creation, and a small minority change constantly. That shape means the answer is rarely a single batch interval for everything — it is a batch for the stable bulk and change-triggered recomputation for the volatile minority, which is cheaper than either policy applied uniformly.

The third question is where most candidate features fall over, and it is worth attacking rather than accepting. Personalisation is often separable: compute the expensive shared part in batch — the summary, the extracted structure, the embedding — and apply the cheap personal part at request time with ordinary code. That decomposition converts an ineligible feature into an eligible one and is the single most useful move in this pattern.

Coverage: the arithmetic

The obvious objection is that you pay for answers nobody reads. Here is when that is fine.

  N   items you would precompute
  r   fraction of them that ever get read in a period
  C   cost of one computation at request-time rates
  B   cost of one computation at batch rates      (B <= C)
  V   value of removing the wait, per read

  precompute everything:   N * B
  compute on demand:       N * r * C            (+ N * r * V of felt latency)

  precomputing everything wins when:   N * B  <  N * r * C + N * r * V
                              <=>      B      <  r * (C + V)

  With B = C (no batch discount) and V ignored, this is just B < r*C,
  i.e. it pays only when the read rate exceeds 1 -- never. So the case for
  precomputing EVERYTHING rests entirely on the batch discount and on the
  value of the removed wait. Be honest about V; if the feature is a nice-to-
  have that nobody waits for, V is near zero and the maths says no.

  Which is why the usual answer is not "everything":
     rank items by P(read), precompute the top slice, compute the rest on
     demand. Coverage of the top decile often captures most reads, because
     access to almost any catalogue is heavily skewed -- but measure YOUR
     skew rather than assuming it.

The practical procedure follows from the last block. Take your access logs, sort items by read frequency, and find the point where the cumulative read share flattens. Precompute above the line, compute below it on demand and write the result into the same store — so a miss populates the cache and the two mechanisms converge over time.

One caveat about the ranking, which is a genuine trap rather than a detail. Read frequency measured on a feature that does not exist yet is measured on the old behaviour, and precomputing changes what people read: an item that is instant gets opened, and an item that took eight seconds did not. So the first ranking is a starting point rather than an answer, and the coverage line should be recomputed a few weeks after launch against the new access pattern. Teams that set the line once tend to find, much later, that they are precomputing yesterday’s popular items and serving today’s on demand.

Keeping it fresh

A precomputed store has the same invalidation problem as a cache and one extra: the job that fills it can silently stop.

• Recompute on change, not on a schedule, where you can. If the source record is written by your own code, enqueue the recomputation from the write path. Nightly rebuilds of everything are the fallback for sources you do not control, and they scale worse every month.
• Store the inputs the answer depended on. Model version, prompt version, source revision — alongside the answer, not in a separate table. Serving code can then decide whether an entry is still valid, and a prompt change becomes a targeted rebuild rather than a full one.
• Alert on the age of the oldest entry, not on job success. A job can succeed while processing nothing. The metric that catches a stalled pipeline is the age of the stalest item in the store, and it catches every variant of the failure with one alert.
• Rebuild incrementally after a prompt change. A change that invalidates the whole store is a large, expensive batch; run it in priority order so the most-read items are correct within minutes rather than after the tail finishes.

The hybrid that usually wins

Few features are purely one thing. The shape that tends to survive contact with production has three tiers, and it is worth designing for directly rather than arriving at by accident.

request
   |
   +-- precomputed?  -> serve from store            (microseconds, already paid)
   |
   +-- computable now within budget?
   |        -> compute, serve, AND write to the store
   |           (a miss is a chance to populate; the two tiers converge)
   |
   +-- otherwise
            -> serve the deterministic fallback now,
               enqueue for the next batch,
               and tell the user it will be ready rather than spinning

The third branch is the one that makes this a design rather than an optimisation. It requires the feature to have a non-AI path and a notion of “not ready yet”, which is a product decision that must be made early — and which is the same decision the progressive-enhancement argument asks for on entirely separate grounds. The mechanics of the queue and of getting the result back to a waiting user are covered elsewhere; what belongs here is the decision that the feature has a batch tier at all, because that decision constrains the interface.