Is Inference Profitable? The Model, Not the Rumour

The question is asked as though the answer were a number somebody is hiding. It is better understood as a formula with one dominant variable, and once that variable is visible the arguments on both sides turn out to be about different values of it.

The unit of production is an hour, not a token

A serving business does not buy tokens. It buys accelerator-hours — through a lease, a cloud contract or a depreciating purchase — and those hours cost the same whether or not anything is running on them. Tokens are what it manages to extract from an hour and sell. Every interesting property of the business follows from that mismatch between a fixed input and a variable output.

This is the same cost structure as an airline seat or a hotel room, and the same conclusions apply: capacity is perishable, an unsold unit is gone forever, and any revenue above the tiny marginal cost of running a machine that is already powered on improves the day. It is emphatically not the cost structure of classic software, where the marginal unit is free and the fixed cost is engineering.

The margin, written down

revenue per accelerator-hour = p * T * U
cost    per accelerator-hour = H

gross margin = (p * T * U - H) / (p * T * U)

  p   realised price per token, after discounts and
      free tiers, not the list price
  T   tokens the server can produce per hour at full load
  U   utilisation: the fraction of capacity actually sold
  H   fully loaded cost of the hour

Everything a serving business does maps to one of those four letters. Better kernels and quantisation raise T. Committed-use discounts and owning rather than renting lower H. Batch tiers and spot capacity raise U. Enterprise contracts raise p. There is nothing else in the business.

Utilisation is the whole answer

Hold everything else fixed and vary only U. All figures below are assumptions chosen to be round:

Assume H = $2.00 per accelerator-hour
       T = 1,000,000 tokens per hour at full load
       p = $3.00 per million tokens realised

full-load revenue = 1,000,000 * 3.00 / 1e6 = $3.00 / hour

U = 0.90 :  revenue $2.70   cost $2.00   margin  +26%
U = 0.67 :  revenue $2.00   cost $2.00   margin    0%
U = 0.40 :  revenue $1.20   cost $2.00   margin  -67%

break-even utilisation:  U* = H / (p * T) = 2.00 / 3.00 = 0.67

Same hardware, same price list, same software; the business is comfortably profitable, exactly break-even or losing two thirds of its revenue depending on one operational number that no price list contains. Any confident claim about whether inference makes money is a claim about U, usually without knowing it.

And U is hard, for reasons that are structural rather than managerial. Demand is diurnal and bursty while capacity is bought in whole machines. Capacity must be provisioned for peak or requests queue. A model that has just been superseded strands the capacity dedicated to it. Reserved capacity for a large customer that under-uses it is idle you are paying for. This is why batch tiers and long-horizon asynchronous jobs are discounted so heavily: they are worth real money because they fill the troughs, and a customer with latency tolerance is selling something genuinely valuable back.

The latency promise is a cost

T and per-request latency trade against each other directly. Larger batches raise total throughput and lengthen the time any individual request waits. So a provider promising a fast interactive experience is choosing a lower T, and a lower T raises the break-even U* because the same hour now yields fewer sellable tokens.

U* = H / (p * T)

Halve T for a tighter latency SLO, at the same p:
  U* doubles.

Which is why the same tokens are sold at different
prices depending on when they must arrive.

Read a price list with that in mind and the tiers stop looking arbitrary. A batch tier, a priority tier and a provisioned-capacity tier are three different points on the same trade, sold to customers with different tolerances. None of them is a discount in the retail sense; each is a different product with a different cost.

The same trade explains why serving a model at very long context, or one that emits a great many reasoning tokens, is priced the way it is. Both consume more of the hour per request — long context through memory occupied per sequence, which limits how many requests fit in a batch at all, and reasoning through sheer decode time. Either reduces the number of billable requests an hour can carry, which is a T effect rather than a pricing decision, and it is why capacity for long-context traffic is scarcer than a price list suggests.

It is three questions, not one

“Is inference profitable” conflates businesses with different cost structures, and an answer for one says nothing about another.

Note what follows from the third row: the more transparent the underlying prices, the thinner an intermediary’s achievable spread, and the more it has to earn its place on something other than price. That is a general property of intermediation and it applies here exactly as it does anywhere else.

What would actually settle it

This page will not tell you the current answer, because the inputs are mostly undisclosed and anything asserted here would be an invention with a short shelf life. It can tell you exactly what to look for, and the list is short:

• Segment-level gross margin in a filing, with the segment defined narrowly enough that inference is not blended into something else. Blending is the usual reason a number is unusable.
• The depreciation life applied to accelerators. It is disclosed in accounting policy notes, it moves H directly, and it is a judgement rather than a fact — see why the schedule is the underrated variable.
• Any utilisation proxy. Rarely disclosed directly. Queue times, availability of provisioned capacity, and how aggressively batch tiers are discounted are all indirect readings of it.
• Whether training is above or below the gross margin line. Two companies with identical operations can report very different gross margins on this choice alone.

When you find any of those, write down the date and the source next to it. A margin figure without a period attached is not evidence about a business whose H, T and p all moved during the period being discussed.