Self-Hosting vs API: The Real Break-Even Point

The self-hosting argument is usually conducted in volume — “above so many tokens a month it pays for itself”. That framing is wrong, and swapping it for the right one changes the answer for most teams.

The comparison people make, and why it fails

The standard calculation takes a GPU instance’s hourly price, multiplies by 730 hours, divides by the tokens the instance can theoretically produce in a month, and compares the result with an API price. It produces an encouraging number and it is wrong in two ways that both push the same direction.

First, it assumes the accelerator is busy all month. It will not be: you must provision for peak and you are billed through the trough. Second, it omits the people. Someone deploys the serving stack, upgrades it, handles an out-of-memory crash at 3am, benchmarks the next model, and maintains the autoscaler. Those hours are the term that most often decides the outcome, and they are the term always missing from the blog post.

Cost per million tokens from a GPU-hour

Start with the honest unit. Let H be the hourly price of the instance (all of its accelerators, not one), and tps the sustained output tokens per second it achieves across all concurrent requests — the aggregate throughput under batching, not the single-stream rate, which is several times lower and is the number people accidentally use.

cost per million output tokens, at full utilisation:

  C_full = H * 1e6 / (tps * 3600)

Worked, with both inputs stated as assumptions — substitute the current list price for the instance you would actually rent, from the cloud’s own pricing page, and a throughput figure you have measured on your own model and sequence lengths rather than one from a vendor benchmark. Take H = $10.00/hour and tps = 1,500:

C_full = 10.00 * 1e6 / (1500 * 3600)
       = 1e7 / 5.4e6
       = $1.85 per million output tokens

Against an API price of, say, $3.00 per million output tokens, that looks like a 38% saving and the meeting ends. It should not, because C_full is a number you will never pay.

Utilisation is the whole variable

You rent the instance by the hour whether or not it is working. Let u be utilisation — the fraction of the rented capacity you actually consume, averaged over the month.

C_actual(u) = C_full / u

Break-even against an API price P:

  u*  =  C_full / P

With C_full = $1.85 and P = $3.00:
  u* = 0.62   -- you need 62% sustained utilisation to break even

Now ask what u can be. For interactive traffic you provision for peak, so utilisation is bounded above by the ratio of mean load to peak load. A product with a working-day traffic shape and a 4:1 peak- to-mean ratio cannot exceed 25% on a fixed fleet, which puts C_actual = 1.85/0.25 = $7.40 per million — nearly 2.5× the API price — before anyone has been paid to look after it.

Autoscaling helps and does not rescue it, because model servers have cold starts measured in minutes: loading tens of gigabytes of weights onto an accelerator is not a container start. You end up keeping warm headroom, and warm headroom is idle capacity by another name.

The exception, and it is a real one: batch workloads can approach u = 1. If you have a queue of work and no latency requirement, you can keep the accelerator saturated by construction — which is the same economic fact that lets providers offer a discount for asynchronous requests. Self-hosting for offline enrichment and self-hosting for a chat endpoint are different decisions with different answers.

Folding ops into the hourly rate

The pleasant surprise in this model is that operational effort does not need a separate, arguable line. Monthly ops cost divides into the same denominator as everything else, so it is exactly equivalent to an increase in the hourly rate:

H_eff = H + F / 730

  F  monthly ops cost in dollars (engineer-hours * loaded rate,
     plus monitoring, registry, storage for weights, egress)

Then everything above holds with H_eff in place of H.

Worked, continuing the example with an assumed 20 engineer-hours a month at a loaded $120/hour:

F      = 20 * 120 = $2,400 / month
H_eff  = 10.00 + 2400/730 = 10.00 + 3.29 = $13.29 / hour

C_full = 13.29 * 1e6 / (1500*3600) = $2.46 per million
u*     = 2.46 / 3.00 = 0.82

Break-even utilisation moves from 62% to 82%.

Eighty-two percent sustained utilisation on interactive traffic is not a stretch goal, it is arithmetically unavailable. That is the honest headline of this page: the break-even is a utilisation number, and ops effort is what moves it out of reach. Twenty hours a month is also a mild assumption — the first month is far more, and every model upgrade is a project.

Two refinements if you want the model to be complete. Input tokens are much cheaper to serve than output tokens, so a fair comparison against a blended API price should account for your input:output ratio rather than pricing everything at the output rate. And reserved or spot pricing changes H substantially — spot in particular can halve it, at the cost of interruptions that an inference server with a five-minute warm-up handles badly.

Where each side genuinely wins

Cost is not the only axis, and being straight about the others is what makes the cost analysis credible.

• Self-hosting wins on data residency and processing guarantees you can prove to an auditor; on workloads with high, steady, batchable volume where u can genuinely approach one; on custom or fine-tuned weights nobody hosts; on freedom from rate limits and deprecation notices; and on predictable unit cost at very large scale, where the fleet is large enough that peak-to-mean smooths out.
• The API wins on everything below high sustained utilisation; on spiky and interactive traffic; on access to frontier models you cannot obtain the weights for; on the ability to change model with a config value; and on not owning an on-call rotation for an inference server.

The hybrid is common and is not a compromise: serve the steady batchable floor on your own hardware at high utilisation, and burst the interactive peak to an API. It is the only configuration in which both sides of the comparison are operating in the regime where they are cheap.

Whichever way it goes, run the numbers with u as an explicit variable and show the sensitivity, because u is the input everybody is most optimistic about and the one that decides the answer.