Forecasting LLM Spend as You Grow

Total spend is the wrong metric to forecast. It rises when the product succeeds, which tells you nothing. Forecast cost per active user, and the total falls out of the growth plan you already have.

The chain

Inference spend decomposes into a product of five terms, each of which someone in the company can estimate or measure:

monthly_spend = U * a * s * r * c

  U  monthly active users of the product
  a  share of them who touch the AI feature at all
  s  AI sessions per engaged user per month
  r  model requests per session
  c  mean cost per request (from your per-request model)

and the number to actually track:

  cost_per_engaged_user = s * r * c

The reason to keep it as a chain rather than a single dollars-per-user figure is that each term has a different owner and a different lever. a and s move with product decisions — putting the feature behind a menu versus on the home screen changes a by a factor, not a percentage. r moves with architecture: adding a verification pass doubles it. c moves with prompt and model choices. When the forecast is wrong, the chain tells you which team’s assumption was wrong.

A worked forecast

Every input below is an assumption, chosen to be plausible rather than observed:

U = 20,000        monthly active users
a = 0.35          engage with the AI feature
s = 6             AI sessions per engaged user per month
r = 4             requests per session
c = $0.0055       mean cost per request

engaged users        = 20,000 * 0.35 = 7,000
requests per month   = 7,000 * 6 * 4 = 168,000
monthly spend        = 168,000 * 0.0055 = $924

cost per engaged user = 6 * 4 * 0.0055 = $0.132 / month

Two of those five terms are worth extra scepticism when you fill them in. a is almost always overestimated early, because the people building the feature use it constantly and everyone else has not found it; measure it rather than assuming, and expect it to be a third of what the team guesses. s is the term that a successful product moves most, and it moves in steps rather than smoothly — a notification, a new entry point, or an onboarding change can double it in a week without anything about the inference changing at all.

Thirteen cents per engaged user per month. That single number is what you carry into a margin calculation, what you compare against your price, and what should be flat on a dashboard while everything else grows. If it is not flat, the forecast is broken and one of the next four things is why.

Four ways the linear model breaks

1. Conversations get longer, and history is resent

If each turn resends the whole conversation, turn n pays for the n−1 turns before it. The cost of a k-turn session is proportional to k(k+1)/2, not to k.

k = 4  ->  4*5/2  = 10 units
k = 8  ->  8*9/2  = 36 units      2x the turns, 3.6x the cost

Retention improves k. So the better your product gets at keeping people in a conversation, the faster this term grows — success and cost are coupled through a quadratic. A sliding window plus a rolling summary restores linearity and is the standard fix.

2. The retrieval corpus grows

More documents usually means more chunks retrieved, longer chunks, or both, and every one of them is input tokens on every request. Track T_in per request over time; if it drifts upward, c is rising underneath a forecast that assumed it constant.

3. Agent loops multiply r

A feature that becomes agentic does not increase r by a percentage; it multiplies it by the step count. Going from one call to an average of six steps is a 6× on the whole forecast, and it arrives in a single release. Any roadmap item containing the word “agent” needs its own line in the forecast.

4. Usage has a heavy tail

Users do not consume the mean. If the top q fraction of users accounts for share s of tokens, then the average user in that group costs s/q times the overall mean:

heavy_user_multiple = s / q

If the top 5% of users generate 40% of tokens:
  0.40 / 0.05 = 8x the mean user

At $0.132 mean, a top-5% user costs about $1.06 / month.

Measure q and s for your own product; they are a group-by over a month of usage. The shape matters because the forecast is fine while the mix is stable and breaks the moment your growth channel starts delivering a different kind of user — an enterprise pilot, an integration partner, a viral thread — and the tail thickens.

Scenarios, not a single number

A point forecast will be wrong, and presenting one invites an argument about the wrong thing. Present three, varying only the two or three terms your sensitivity check says dominate.

                low        base       high
U              15,000     20,000     35,000
a                0.25       0.35       0.50
s*r               18         24         40
c            $0.0040    $0.0055    $0.0080

spend          $270       $924     $5,600

The high case is 6x the base, from individually modest
changes, because the terms multiply.

The discipline is to vary only what you are genuinely uncertain about, and to say out loud what each column assumes. A high case that silently combines the most pessimistic value of every term is not a scenario, it is a worst case, and presenting it as a forecast destroys the credibility of the other two columns. Name the story behind each: “low” is the current growth rate continuing with no product change; “high” is the enterprise pilot converting and the agent feature shipping in the same quarter.

That multiplication is the point of the exercise. Four assumptions each optimistic by a factor of 1.5 produce a 5× surprise, which is the normal way a bill becomes a crisis. It is also the argument for a hard cap set from the high case rather than the base one.

Keeping it honest

• Re-fit monthly, from logs. Every term is measurable. A forecast that has not been compared with the actual is a wish.
• Alert on cost per engaged user, not on total spend. Total spend rising with users is expected; cost per user rising is a regression and needs an explanation.
• Re-forecast on any change to r. Prompt edits change c gradually; architecture changes change r in steps. The steps are what break budgets.
• Keep the model in the repository. A spreadsheet in someone’s drive is not re-runnable. Twenty lines of code against your usage table is.
• Forecast the second-order costs too. Vector storage, logs and evaluation runs scale with the same terms and are routinely left out.