When Local Inference Is Cheaper Than an API

The comparison is usually made badly, because one side is quoted per million tokens and the other as a purchase. Convert the purchase into the same unit and the argument resolves itself — and the deciding term turns out not to be the hardware at all.

Get both sides into one unit

An API price is currency per million tokens. To compare, express your own machine the same way. That requires three quantities: what the box costs per hour to own and run, how many tokens it can produce per hour, and what fraction of the hours it is actually producing.

The third one is where nearly every published comparison goes wrong. A GPU that is idle still costs its amortisation, and a machine bought for one developer is idle the overwhelming majority of the time.

One preliminary, because it invalidates the whole exercise if you skip it: the two sides must be serving comparable capability. Comparing a small open model on your own hardware against a frontier API is comparing the price of two different things, and it will always flatter the local side. Either compare against the hosted price of the same open model, or accept that you are making a quality trade as well as a cost one and say so explicitly. The arithmetic below assumes you have already fixed the model.

The formula

hourly_cost = amortised_capital_per_hour
            + (power_kw * electricity_per_kwh)
            + operational_per_hour

amortised_capital_per_hour = purchase_price / (life_years * 8760)

tokens_per_hour = sustained_tokens_per_sec * 3600 * utilisation

cost_per_million = hourly_cost / (tokens_per_hour / 1e6)

Every input is something you either know or can measure. sustained_tokens_per_sec comes from running llama-bench or a serving benchmark on the actual hardware with the actual model — not from a published figure. utilisation is generated tokens divided by tokens the box could have generated, over a real week. power_kw is the wall draw under load, which is meaningfully higher than the GPU’s own figure once the rest of the system is counted.

Note what is not in the formula: nothing about how good the model is. This computes the price of a token, and a token from a smaller model is not the same good as a token from a larger one. Compare like with like or the exercise is decorative.

Working it through

Take a single-GPU box serving one mid-size open model, and suppose your measurement gives 30 tokens per second sustained for a single stream. Call the all-in hourly cost H, in whatever currency you computed it in.

one developer, one stream, busy 40% of an 8-hour day
  utilisation = 0.40 * (8/24) = 0.133
  tokens_per_hour = 30 * 3600 * 0.133 = 14,400
  cost_per_million = H / 0.0144 = 69 * H

the same box, batching server, 10 concurrent users,
aggregate 250 tok/s, busy 30% of the whole day
  tokens_per_hour = 250 * 3600 * 0.30 = 270,000
  cost_per_million = H / 0.270 = 3.7 * H

the same box, saturated batch job, aggregate 400 tok/s, 95% busy
  tokens_per_hour = 400 * 3600 * 0.95 = 1,368,000
  cost_per_million = H / 1.368 = 0.73 * H

The hardware did not change. The cost per million moved by a factor of ninety-four. Look up a current per-million price for a comparable open model, divide by your H, and you have the threshold: if the quotient is above 69 in the first scenario, self-hosting is cheaper; below it, it is not. For most single-user setups the quotient is not close, and the machine is being justified by something other than cost.

Utilisation is the whole game

Two structural facts drive that table, and they are worth stating plainly because they invert the usual intuition.

• An API bills only for tokens; a box bills for time. Idle API capacity costs nothing. Idle GPU costs full amortisation. All of the economics of self-hosting is the fight to keep the card busy.
• Batching is nearly free throughput. Generation is memory-bandwidth bound, so the weights streamed for one token can serve many sequences at once. Going from one stream to ten concurrent streams multiplies aggregate tokens per second by far more than it multiplies cost — which is why a batching server rather than a single-stream runtime is the difference between the second row and the first.

The practical consequence: if you are self-hosting for cost reasons, the first engineering task is not choosing hardware. It is making sure the hardware you have is shared, queued and batched.

The terms people leave out

• Your time. Setup, upgrades, driver breakage, model updates, an on-call path when it dies at 2am. At any professional hourly rate this is frequently the largest term in the model and it is almost never included.
• Redundancy. One box is a single point of failure. Costing a production service honestly means costing two, or costing the fallback path.
• Peak versus average. Hardware must be sized for peak concurrency; the cost model above divides by average. The larger the ratio, the worse self-hosting looks.
• Residual value and obsolescence. The amortisation period is a guess about how long the card stays adequate, and capability per unit of memory has been moving quickly.
• On the API side: caching and batch tiers. Discounted asynchronous processing and cached prompt prefixes can move the per-million figure substantially for the workloads that qualify. Compare against the price you would actually pay.

When each side wins

Self-hosting wins on sustained high utilisation — continuous batch processing, an embedding pipeline over a large corpus, a classifier in a hot path — and on any workload where the data may not leave your control, where the cost model is beside the point. It also wins when the hardware exists already and its amortisation is sunk into something else.

An API wins on spiky, low-average traffic, on anything needing frontier capability, and on every early-stage system where you do not yet know your volume — which is nearly all of them.

The sequence that avoids the expensive mistake is to start hosted, instrument the traffic, and re-run this calculation once you have a month of real token counts and a measured utilisation figure. By then the answer is arithmetic rather than argument, and the case for buying hardware — if there is one — will be obvious enough to make without a spreadsheet fight. Buying first and measuring afterwards inverts that, and the machine tends to acquire justifications after the fact.