Why “Free” Local Inference Still Has a Real Cost

“Once the hardware is paid for, inference is free” is not wrong so much as incomplete. It is a claim about marginal cost, made in a situation where marginal cost is the smaller term, and the arithmetic that shows this is short enough to do here.

The claim, stated fairly

The strong version of the argument is genuinely strong. A hosted model charges per token, forever, at a price somebody else sets. A local model charges nothing per token, and no invoice grows with usage. For a workload that runs constantly, that difference compounds, and the people who make this argument are usually the people running such a workload.

The claim becomes misleading when it is transplanted to a personal machine used a few hours a day. There the fixed cost is not amortised across enough tokens to disappear, and two costs that the framing treats as zero — electricity and idle — are not zero. The position of this page is that local inference is best understood as converting a variable cost into a fixed one, and that whether the conversion is worth making is a utilisation question with an arithmetic answer, not an ideological one.

The electricity, worked

Two inputs, both published. The US Energy Information Administration’s Electric Power Monthly gives the average residential retail price of electricity as 18.44 cents per kilowatt hour for May 2026 in table 5.6.A. Board power is published by the GPU vendor per card; take a card rated at 450 W and assume it draws its full rating under a sustained generation load, which is the pessimistic end.

Card under load:  0.450 kW x $0.1844/kWh = $0.0830 per hour

Cost per million output tokens, as a function of the
sustained generation rate T (tokens per second):

  tokens per hour   = T x 3600
  $ per Mtok        = 0.0830 / (T x 3600 / 1e6)
                    = 0.0830 / (0.0036 x T)

  T =  20 tok/s ->  $1.15 per million tokens
  T =  50 tok/s ->  $0.46 per million tokens
  T = 100 tok/s ->  $0.23 per million tokens

Those three results follow from that formula at those three rates. They are not measurements of any particular model on any particular card, and the rate T is exactly the number that no source publishes for your combination — measuring tokens per second on your own machine is a twenty-second job and gives you the only value of T worth substituting in.

The interesting thing about those numbers is their size. Per million tokens, the electricity alone lands in the same range as the cheapest hosted small models, before any hardware has been paid for. That should not be surprising: hosted providers run the same silicon and pay commercial rather than residential rates, and they amortise the card across a far higher duty cycle than a desktop achieves.

The hardware, amortised

The fixed cost only vanishes if you refuse to divide it. Divide it and the picture is clear. Take a machine at $1,200, a three-year useful life before you would want to replace it anyway, and, generously, two hours a day of actual generation:

capital per year      = $1,200 / 3        = $400.00
hours of use per year = 2 x 365           = 730 hours
capital per hour used = $400 / 730        = $0.548

total per hour of use = $0.548 + $0.083   = $0.631
share that is power   = 0.083 / 0.631     = 13%

Under those assumptions the electricity is about an eighth of the cost, and the thing the “free” framing dismisses entirely is seven eighths of it. Push the usage to eight hours a day and capital per hour falls to about $0.137, power becomes 38% of the total, and the local option starts to look like what its advocates describe. The variable that decides the argument is utilisation, and nothing else in the calculation is close.

This is also the answer to why the same argument reaches opposite conclusions for two people who both believe they are reasoning correctly. A developer running batch jobs overnight and an occasional chat user are separated by an order of magnitude in duty cycle, which moves the crossover by an order of magnitude. The general comparison against per-token pricing is worked in local versus API cost; this page’s contribution is the reminder that the denominator is the load-bearing part.

Idle is the line item people miss

A machine that exists to answer questions on demand has to be on. A model kept resident so the first request is fast has to stay loaded. Neither of those is free either:

Idle draw 60 W, running continuously:

  0.060 kW x 8,760 h/yr        = 525.6 kWh per year
  525.6 kWh x $0.1844/kWh      = $96.92 per year

  ... which at 730 hours of actual use per year adds
  $96.92 / 730 = $0.133 to every hour of use.

That figure is larger than the electricity consumed by the generation itself in the two-hours-a-day scenario. It is invisible because it does not correlate with anything you did; it appears on a bill next month, undifferentiated. Keeping a model resident raises the floor further, and how much depends on the card and the driver’s power state behaviour — idle power with a model loaded is the specific treatment. The alternative, unloading between requests, converts the cost into cold-start latency instead, which is a real trade and not an elimination.

Where local genuinely wins

The arithmetic above says local inference is rarely the cheaper option at low volume. It does not say local inference is a bad idea, because three of its advantages are not priced in tokens at all.

• Data that does not leave. If the reason for running locally is that the input cannot be sent anywhere, the cost comparison is not the deciding calculation and this whole page is a footnote to it.
• No rate limit and no deprecation. A local model answers at 3am during an incident, does not return 429, and does not change behaviour because a vendor shipped a new checkpoint under the same name. That stability has a value that is real and hard to express per token.
• A latency floor you own. Network round trip and queueing at a provider are outside your control. On a loaded local model, time to first token is a property of your machine.

What the arithmetic does rule out is the specific claim that local inference is free. It is prepaid, which is a different thing, and the prepayment is only a good deal at volume you should check rather than assume.