Battery and Thermal Cost of On-Device Inference: Joules Per Token

The energy a generated token costs is dominated by moving weights out of memory, not by the arithmetic performed on them. That single fact makes the cost derivable: multiply the bytes that must move per token by the energy it takes to move a byte. Then measure it, because the second factor is the one you should not take on trust.

Energy follows bytes moved, not operations

On modern silicon, an arithmetic operation is extraordinarily cheap relative to fetching its operands from off-chip memory. Mark Horowitz’s 2014 ISSCC keynote “Computing’s Energy Problem (and what we can do about it)” put the gap at roughly four orders of magnitude at the process node of the day: an 8-bit integer addition around 0.03 pJ, a 32-bit DRAM access around 640 pJ. The absolute numbers have improved since and vary by memory generation; the ratio has not closed.

For token generation the consequence is direct. Producing one token requires reading every active weight once — the arithmetic per weight is a single multiply-accumulate. So the energy per token is approximately the energy to read the model once, and the model is the overwhelming majority of the bytes.

This also explains the entire economics of quantisation on a battery device. Halving bytes per weight halves the bytes moved per token, and therefore approximately halves the energy per token — a far larger effect than any arithmetic saving.

The derivation

bytes_per_token ≈ weight_bytes + kv_bytes_read

energy_per_token ≈ bytes_per_token × E_byte

Assumptions, stated:
  model                3B parameters
  quantisation         4-bit grouped, 0.58 bytes/weight
  weight_bytes         3e9 × 0.58 = 1.74e9 bytes
  kv read per token    small at short context; ignored here,
                       and it grows with context length
  E_byte               the unknown — see below

Take the Horowitz figure as a deliberate upper bound: 640 pJ per 32-bit access is 160 pJ per byte, or 1.6e-10 J.

Upper bound, at 160 pJ/byte:
  1.74e9 × 1.6e-10 = 0.278 J per token

If mobile LPDDR is ten times better than that 2014 figure:
  1.74e9 × 1.6e-11 = 0.028 J per token

A 300-token answer therefore costs somewhere between
  8.4 J  and  83 J  of memory-system energy alone.

A range spanning an order of magnitude is not a failure of the method, it is an honest statement of what is knowable without measurement. What the derivation gives you that a benchmark does not is the structure: energy scales linearly with model bytes, linearly with tokens generated, and is essentially unaffected by how fast you generate them. Those three relationships are what you design against, and they are true regardless of which end of the range your device sits at.

Where the derivation is weakest

• Caches break the “read it all” assumption. An accelerator with substantial on-chip memory may keep part of the model resident across tokens, and on-chip access is far cheaper. For models much larger than on-chip memory — which is all of them — most traffic is still off-chip, but the fraction is not 100%.
• Prefill is a different regime. Reading a long prompt processes many tokens per weight fetch, so it is arithmetic-bound rather than bandwidth-bound. Prompt energy per token is much lower than generation energy per token. Budget them separately.
• The KV cache term grows. At short contexts it is noise. At several thousand tokens it is a significant additional read per token, using the per-token cache arithmetic in the phone memory budget.
• The system is not only the memory. The display, the radio and the CPU cores coordinating the work all draw power during the generation. For a short answer on a screen the user is watching, the display can be the larger term.

Measuring it in your own app

Android exposes the battery’s coulomb counter, which makes this a direct measurement rather than an estimate. Read the accumulated charge before and after a fixed workload and convert:

val bm = context.getSystemService(Context.BATTERY_SERVICE) as BatteryManager

// Accumulated battery charge, in microamp-hours.
fun chargeUAh(): Long =
    bm.getLongProperty(BatteryManager.BATTERY_PROPERTY_CHARGE_COUNTER)

val before = chargeUAh()
val tokens = generateFixedWorkload()      // e.g. 20 answers of 300 tokens
val after  = chargeUAh()

val deltaUAh   = (before - after).toDouble()
val nominalV   = 3.85                      // pack nominal voltage
val joules     = deltaUAh * 3.6e-3 * nominalV
val jPerToken  = joules / tokens

// 1 µAh = 3.6e-3 coulombs;  joules = coulombs × volts

The protocol around that reading is what makes it trustworthy:

1. Device off charge, screen at a fixed brightness, aeroplane mode on unless the network is part of what you are measuring.
2. Measure a baseline first: the same duration with the app open and idle. Subtract it. What you want is the marginal cost of inference, not the cost of the phone being on.
3. Run long enough for the delta to be many times the counter’s resolution. A few minutes of continuous generation, not one request.
4. Let the device reach thermal steady state before you start counting, and note that the sustained rate — not the first-ten-seconds rate — is the one that pairs with this energy figure.
5. Repeat on the oldest device you support. Energy per byte and sustained clocks both differ across generations, and the old device is where the complaint comes from.

Turning joules into something a user feels

Joules are not a product decision. Convert them:

battery_joules = capacity_mAh × 1e-3 × 3600 × nominal_V

Worked: a 4,000 mAh pack at 3.85 V
  4.0 × 3600 × 3.85 = 55,440 J

One per cent of that battery = 554 J

If a 300-token answer costs 7 J (measured, not assumed):
  554 / 7 ≈ 79 answers per one per cent of battery

That is the sentence to put in front of a product decision: not “inference uses battery”, but “at our measured cost, a heavy user generating fifty answers a day spends under one per cent of their battery on the model” — or, if the measurement comes out differently, “six per cent”, which is a different product.

Do the same arithmetic for the alternative. A hosted call costs the device a radio transmission and a few seconds of screen time, which is often the cheaper option in joules even though it feels less efficient. Local inference wins on privacy, offline capability and marginal cost per request; it does not automatically win on battery, and claiming otherwise without the measurement is exactly the kind of assertion this cluster exists to replace.

Heat is the same energy, arriving as a complaint

Every joule you spend leaves as heat. A phone dissipates it through its case, which is why sustained inference has two user-visible consequences that battery percentage does not capture: the device gets warm in the hand, and the system reduces clocks to stop it getting warmer.

The relationship between the two is worth stating precisely. Battery drain depends on total energy — joules — and so is proportional to how many tokens you generated in total. Heating depends on power — joules per second — and so is proportional to how fast you generated them. A feature that produces a thousand tokens over ten minutes and one that produces a thousand tokens in one minute cost the same battery and feel completely different.

That gives you a lever that is invisible if you only think about energy: spread the work. Background processing of a queue can be deliberately paced — generate, pause, generate — so that average power stays below the point at which the governor intervenes. The total takes longer and costs the same joules, and the device stays cool and never throttles, which means the sustained rate is the boost rate rather than the reduced one. For interactive work the opposite is true and you want the burst, so the two paths should be paced differently rather than sharing one implementation.

Both platforms expose a coarse thermal-state signal. Subscribe to it, and define what your feature does at each level in advance: full behaviour, then shorter outputs or a smaller model, then defer to a server, then disable and say so. Deciding this while the device is already hot produces a worse answer than deciding it in a design review.

What actually reduces the cost

• Smaller weights. Linear in bytes moved, therefore linear in energy. Going from 8-bit to 4-bit weights roughly halves the dominant term. This is by far the biggest lever.
• Fewer output tokens. Also linear. “Answer in two sentences” is a battery optimisation with the same standing as a latency one.
• A smaller model for the easy cases. Routing short classification work to a 300M model instead of a 3B one is a ten-times energy reduction on that request, not a marginal one.
• Not running at all. Cache results. Debounce. Do not re-summarise a document the user has not changed. The cheapest token is the one you did not generate, and this is the lever teams reach for last.
• Batching, where the interface allows it. Generating for several inputs in one pass amortises the weight read across them. It rarely applies to interactive use and applies very well to background processing of a queue.