Cost per Request: Building a Model You Can Forecast With

Most cost models are one multiplication: an average token count times a price. They are wrong in a specific and predictable direction, and the fix is not more decimal places — it is admitting that request sizes are a distribution.

The average request does not exist

Take a support assistant. Some requests are “what are your opening hours” against a short prompt. Some drag forty turns of history plus eight retrieved documents through a model that then writes a page. If the first is 1,200 tokens and the second is 24,000, the mean is meaningless in the same way the average of a mouse and an elephant is: no request looks like it, and the two classes respond to completely different interventions.

The mean is also biased upward by the tail in a way that makes cheap traffic look expensive, which is exactly the mistake that leads to optimising the wrong thing. A model that tells you the mean cost is $0.02 and nothing else cannot tell you whether to shorten the system prompt or cap the conversation length, and those are the only two questions you had.

Split the traffic into classes

Two to five classes is enough, and they should be chosen by what makes them expensive, not by product feature. Useful splits: cached versus uncached prefix; retrieval on versus off; short answer versus long answer; reasoning enabled versus not; first turn versus deep in a conversation.

For each class you need three numbers, all of which you can get from a day of logged usage rather than from a guess: its share of traffic w, its mean input tokens, and its mean output tokens. If you are not logging token counts yet, that is the first thing to fix; the usage object in the response already contains them.

Two boundaries are worth deciding deliberately before you start counting. The first is what a “request” means. If one user action fans out into a retrieval call, a generation and a verification pass, then modelling the generation alone understates the cost by whatever the other two contribute, and the number you actually want is cost per user action. Define the unit as the thing a product person would recognise, and let a class contain several model calls if that is what the architecture does.

The second is whether failed and retried attempts are inside the unit. They should be — you paid for them — but they are usually logged as separate rows and disappear from a naive average. Group by the logical request id rather than by the provider call, and the retry surcharge is included automatically instead of being a correction factor somebody has to remember to apply.

The model

The expected cost of a randomly chosen request is the share-weighted sum of the class costs:

E[cost] = SUM over classes i of
            w_i * ( P_in * T_in_i + P_out * T_out_i ) / 1e6

  w_i     share of requests in class i, sums to 1
  T_in_i  mean input tokens for class i
  T_out_i mean output tokens for class i
  P_in    input price, $ per million tokens
  P_out   output price, $ per million tokens

Worked, with every input labelled as an assumption — prices of $1.00/M in and $5.00/M out, and a traffic mix invented for the example:

class          w      T_in    T_out   cost/req
short         0.70    1,200     150   $0.00195
retrieval     0.25    9,000     400   $0.01100
long thread   0.05   24,000     900   $0.02850

E[cost] = 0.70*0.00195 + 0.25*0.01100 + 0.05*0.02850
        = 0.001365 + 0.002750 + 0.001425
        = $0.00554 per request

Now look at what that decomposition says, because it is the point of splitting at all. The 5% of traffic that is long threads contributes 26% of the expected cost. The 70% that is short contributes 25%. A change that shaves 20% off the short class is worth 5% of the bill; a conversation-length cap that halves the long class is worth 13%. You could not have seen either from a mean.

Getting a p95, not just an expectation

Expected cost tells you the monthly bill. It does not tell you what your most expensive user can do to you, and that is the number a margin calculation and a spend cap both need.

You do not need a fitted distribution for this. Sort a day of logged requests by cost and read the percentile off directly — the empirical quantile is both simpler and more honest than assuming a shape. If you only have the class model, the crude bound is the cost of the most expensive class, and the crude p95 is whichever class boundary the top 5% of traffic falls into. In the example above, the top 5% is exactly the long-thread class, so p95 cost is about $0.0285, roughly 5× the expectation. A 5× ratio between p95 and mean is normal for this kind of workload and is the reason per-request caps get written.

Which input is worth measuring

Every input to the model has an error bar, and most of them do not matter. Find the ones that do by perturbing one input at a time by ±20% and recording how much E[cost] moves. The elasticity is:

elasticity(x) = (% change in E[cost]) / (% change in x)

An input with elasticity near 0.5 halves your error when you measure it properly; one near 0.02 is not worth an afternoon. In the example, T_in of the retrieval class has elasticity around 0.5 — retrieval input is roughly half the bill — while T_out of the short class is under 0.05. That is a direct instruction about where to spend the next hour of work.

Rebuild the model whenever any of three things change: the prompt, the model, or the traffic mix. The first two are deploys and can be guarded automatically; the third happens on its own as users change how they use the product, which is why the model wants a monthly re-fit rather than a permanent home in a slide.