The Energy and Water Cost of Inference

4 min read · updated August 3, 2026

Estimates of the energy in one model query differ by orders of magnitude across credible sources. Most of that spread is not disagreement about physics. It is disagreement about where the system boundary is drawn, and a figure quoted without its boundary is not a figure.

Why the published figures disagree

Before comparing two numbers, establish which of these each one includes. Any of them can change the answer by more than the model choice does:

Which model, and how much output. A short answer from a small model and a long answer from a large reasoning model differ by several orders of magnitude on their own. A single “per query” figure averages over a distribution nobody specifies.
Batch size and utilisation. The dominant engineering term. Serving many requests concurrently amortises the weight read across all of them; the same hardware at low occupancy spends nearly the same power for a fraction of the tokens.
Facility overhead. Cooling, power conversion and distribution, captured as power usage effectiveness. It multiplies everything, and whether a figure includes it is frequently unstated.
Training amortisation. Some analyses divide training energy across expected lifetime queries. Defensible, and it produces a different quantity from marginal serving energy. They are not comparable.
Embodied energy. Manufacturing the accelerators, the building and the power infrastructure. Usually excluded, occasionally included, rarely flagged.
Idle and provisioned capacity. Capacity is held for peak. Charging queries only for the seconds they compute understates the system; charging them for provisioned capacity overstates the marginal query. Both are used.

Building the estimate yourself

The marginal serving calculation is not complicated, and doing it once makes every published figure legible.

Serving side, per accelerator:

  E_per_token  =  (P_device · n_devices · PUE) / R_tokens_per_second

    P_device        average power draw under load, from the spec sheet
                    (below the rated maximum in practice)
    n_devices       accelerators the model is sharded across
    PUE             facility overhead multiplier
    R_tokens_...    TOTAL tokens per second across the whole batch,
                    not per request  ← this is the term that dominates

Per request:

  E_request  ≈  E_prefill(prompt_tokens)  +  E_per_token · output_tokens

  and for a reasoning model, output_tokens includes the thinking
  tokens, which are frequently the majority of them.

The structure of the expression is the lesson. Energy per token is accelerator power divided by aggregate throughput, so anything that raises throughput at fixed power lowers energy per token proportionally. Batching is that lever, and it is why a busy, well-loaded deployment is far more energy-efficient per token than a lightly used one — the opposite of the intuition that heavy use is the problem.

To use it, take the device power and the published throughput for a comparable serving configuration, decide your PUE, and multiply through. The result is marginal serving energy, excluding training and embodied cost, and it should be quoted that way.

Two notes on the training term, which is the boundary choice people argue about most. Training energy is a fixed cost paid once, so the amortised share per request depends entirely on how many requests the model ends up serving — the amortised figure is a claim about lifetime volume as much as about energy, and it should be stated with the assumed volume attached. And the marginal figure is the right answer to “what does one more request cost” and the wrong answer to “what did this capability cost to bring into existence”. Both are legitimate questions. Conflating them produces arguments in which each side is computing something different and neither says which.

Which terms actually move it

Lever	Description
output length	Linear, and the term a user controls. Asking for a concise answer is an energy decision as much as a latency one. Reasoning models can spend far more tokens than they return.
batch size / utilisation	Large and non-linear at the low end. Most of the difference between a well-run and a badly-run deployment is here, not in the model.
active parameters	Drives arithmetic and memory traffic per token. Sparse models decouple this from total size, which is why headline parameter counts predict energy poorly.
precision and quantisation	Lower precision reduces both memory traffic and arithmetic energy, usually the cheapest real reduction available.
prompt caching	Removes repeated prefill for a shared prefix. Where prompts are long and repetitive this is a large share of total work.
routing to model size	Sending easy requests to a smaller model is the largest available reduction for most applications, and the one that requires actual engineering rather than a setting.

Water is two different numbers

Water figures are the most confused part of this discussion because two unrelated quantities are reported under one word.

On-site consumption is water evaporated by cooling. It depends on the cooling design — evaporative systems consume water and save electricity, closed-loop and air-cooled systems do the reverse — and on climate and season. It is a genuine local resource question in a water-stressed region, and it is a design choice rather than an inherent property of computing.

Off-site water is attributed to generating the electricity, mostly thermoelectric cooling at power stations. It usually dwarfs the on-site figure and it depends on the generation mix rather than on the data centre at all. Analyses reporting a striking water number often include it.

Neither is wrong. Reporting one without saying which is. And note the trade: a facility can reduce water by using more electricity, so optimising the two together is a real engineering decision with no free answer.

Per-query is the wrong unit for policy

A per-query figure is useful for personal reasoning and nearly useless for policy. What a grid experiences is aggregate load with a time and location profile — how much power, drawn where, at which hours, and how flexibly. A small per-query number multiplied by enormous volume, placed in one constrained region, is a policy problem that no per-query metric expresses, and a large per-query number in a place with surplus low-carbon generation may not be one at all.

The aggregate and siting questions are a separate subject, covered on the data centre page in this cluster. The one thing to carry across: demand for compute is not fixed, and improvements in efficiency have historically been followed by increases in use, so per-unit efficiency gains cannot be assumed to reduce total consumption. Whether that rebound dominates in this case is an open empirical question and not one this page can settle.

A related caution about the comparison that appears most often: a model query measured against a conventional web search. The comparison is usually made between two quantities computed on different boundaries and from different eras, and the two are not substitutes anyway — one returns links, the other returns an answer, and a user may issue very different numbers of each to accomplish the same thing. If you want the comparison to mean something, fix the task rather than the query, count all the interactions each approach required, and use the same boundary on both sides. Almost no published version of this comparison does all three.