The Economics of Inference: Why Prices Keep Falling

“Inference is getting cheaper” is true and almost useless as stated, because it does not say which of three unrelated things is happening. Separate them and you can tell whether the fall continues, which parts of it you benefit from automatically, and which parts you have to go and collect.

A price is three terms

A provider serving a model on rented or owned accelerators has a price per token that decomposes cleanly:

p = (H / T) * (1 + m)

  p   price per token
  H   fully loaded cost of one accelerator-hour
  T   tokens produced per accelerator-hour, at the latency
      target the product promises
  m   markup over cost

H is the hardware bill amortised into an hour, plus power, cooling, floor space, networking and the people who keep it running. It falls when accelerators get cheaper per unit of throughput, when capacity is plentiful, and when a facility is run better. It rises when any input to it is scarce.

T is the term people underestimate, because it is not a property of the hardware. It is hardware multiplied by everything the serving stack does: batch size, attention implementation, quantisation, how much of the prompt was already cached, and the architecture of the model itself — a sparse model activates a fraction of its weights per token and therefore produces far more tokens per hour than its headline size suggests. Two providers with identical hardware can differ severalfold in T, and that difference is entirely software.

m is the only term that is a decision rather than a fact. It is bounded below by zero over any sustained period and bounded above by what a customer will pay before switching, which is a function of how substitutable the model is.

Taking logarithms makes the structure obvious: ln p = ln H − ln T + ln(1 + m). The three effects are additive in log space, which is another way of saying their percentage changes multiply. That is the whole reason the declines look implausible from the outside.

Why modest movements compound

Take one year and three assumed movements, none of them remarkable on its own:

H falls 15%          H1/H0 = 0.85
T doubles            T0/T1 = 0.50
markup 60% -> 30%    1.30/1.60 = 0.8125

p1/p0 = 0.85 * 0.50 * 0.8125 = 0.345

A 65% fall in the price of the same output,
from three changes any one of which you would
call unexciting.

Nobody had to invent anything. A fifteen percent improvement in the cost of capacity is an ordinary year in a hardware market. A doubling of tokens per hour is a serving-stack release plus a quantisation change. Markup compression is what competition does. Multiply and the headline writes itself.

The same arithmetic tells you the decline is not perpetual, because the three terms have different floors. Markup can compress once and only once — from a healthy margin to zero is a bounded move, and any price below marginal cost is a promotion rather than a trend. T improvements run into physics: decoding one token requires moving the active weights through memory, so throughput per accelerator is bounded by memory bandwidth, and batching harder eventually breaks the latency promise that made the traffic valuable. H is the only term with genuinely open-ended headroom, and it depends on a supply chain nobody in software controls.

Two prices moving in opposite directions

Here is the distinction that most discussions of falling prices skip, and it changes the conclusion completely. There are two different prices and they behave differently:

• The price of a fixed capability level. “A model that scores at least X on my own evaluation set.” This is a fixed good. Its price falls fast, because every year more suppliers can hit that bar and they hit it with smaller, cheaper models.
• The price of the best available model. This is not a fixed good. It is a different product each time, defined by being the most capable thing anyone is selling, and the most capable thing is generally the one somebody spent the most to produce. A series of prices for “the frontier” is a series of prices for different objects, and there is no reason for it to fall.

Both statements can be true at once, and when they are, the popular summary — prices are collapsing — and the contrary observation — the good stuff costs as much as it ever did — are both correct and about different goods.

The practical consequence is a decision about your own requirement. If you can name the capability you need and test for it, the falling price is available to you, but only if you go and take it: you have to re-evaluate cheaper models on a schedule, because nothing moves your traffic for you. If your requirement is “the best available” then you have opted out of the deflation entirely, and you should say so explicitly rather than budget as if prices will save you.

What would stop it

The last one deserves emphasis because it is not a market force at all — it is your own architecture. A price that halves while your agent makes three times as many calls is a cost increase wearing a discount.

Checking it for yourself

If this matters to a budget, do not rely on anyone’s chart, including a chart of provider list prices. Build the only series that answers your question:

• Fix a task and an acceptance bar on your own evaluation set. The bar is what makes successive observations comparable.
• Each quarter, find the cheapest model that clears the bar, and record its cost per completed task — not per token — along with the date and the URL the prices came from.
• Record cached-input and output prices separately, since a change in the ratio between them moves your cost without moving any headline.

That series is quality-adjusted by construction, which is exactly what a per-token chart is not; the index-number problems with the per-token version are worth knowing before you quote one at anybody. And when you compare the two curves you will find they can disagree, for the reasons in the four cost curves.