Why NPUs Use So Much Less Power Than GPUs for On-Device Inference

“NPUs are more power-efficient than GPUs” is true and is almost always argued from numbers that do not support it. This page works out what the published figures do support, derives the rest on the page, and is explicit about the figure nobody publishes.

What is actually published

Start with a single primary document, because it contains the whole problem. Qualcomm’s Snapdragon X Elite product brief (document DCN 87-71417-1 Rev C) lists, for the X1E-84-100 part: a Hexagon NPU at 45 NPU TOPS, an Adreno GPU at 4.6 TFLOPs, and LPDDR5x memory at 8448 MT/s(Qualcomm).

Three observations, and they are the reason most comparisons of this kind are meaningless. The NPU is quoted in integer operations and the GPU in floating-point operations, so the two cannot be divided. Neither carries a stated precision. And the document gives no power figure for either block — there is a package power envelope for the SoC and nothing that attributes watts to the NPU or the GPU individually.

The same holds for the other vendors. AMD’s press release of 6 January 2025 quotes up to 50 TOPS for XDNA 2 without stating the precision in the release (AMD). Intel quotes 48 TOPS for NPU 4 in Core Ultra 200V and 50 TOPS for NPU 5 in Core Ultra Series 3, inside platform totals of 120 and 180 TOPS that also count the GPU and CPU. None of them publishes the power the NPU block draws on its own.

Deriving a bound, not a ratio

Because no vendor publishes per-block power, TOPS-per-watt cannot be read off a spec sheet for any NPU. What you can do is derive a lower bound, by dividing a published peak rate by a published power envelope that is larger than the block’s own draw. Doing it for a discrete GPU, where both numbers appear on one page:

NVIDIA's own RTX 4090 product page lists:
  AI TOPS               1321
  Total Graphics Power   450 W

  1321 TOPS / 450 W = 2.94 TOPS/W

Assumptions, stated:
  - "AI TOPS" carries no precision on that page, and NVIDIA's tensor
    figures are conventionally quoted for low precision and sometimes
    with structured sparsity, either of which inflates it. So 2.94 is
    an UPPER bound on the sustained figure, not a typical one.
  - 450 W is the whole board: memory, fans, VRM, display outputs.
    So it is also a lower bound on the compute-only efficiency.
  These two errors point in opposite directions and do not cancel.

Do the same for a laptop NPU and the arithmetic is even weaker: 45 TOPS divided by a whole-package envelope in the tens of watts gives a number in the same single-digit range, and it is a floor rather than a figure. The conclusion to draw is not “they are similar” — it is that peak-TOPS-over-package-watts is too blunt to separate them, and anyone quoting a clean NPU-versus-GPU TOPS/W ratio has either measured it themselves or made it up. NVIDIA’s figures above are from its own product page (NVIDIA).

Why TOPS-per-watt is the wrong number here

Even a perfect TOPS-per-watt figure would not tell you the energy per token of a language model, because single-stream decoding is not bounded by arithmetic. Every token requires reading the model’s weights out of memory once, and at batch size one there is almost no arithmetic per byte read. The ceiling is memory bandwidth, and it can be derived from the same product brief:

Memory bandwidth, from the published transfer rate:

  8448 MT/s x 128-bit bus / 8 bits per byte
    = 8448e6 x 16 bytes
    = 135.2 GB/s

  (Assumption: a 128-bit LPDDR5x interface. The brief gives the
   transfer rate; the bus width is a platform property — confirm it
   for the exact part before relying on this.)

Weights that must be read per token, for a 3B model at ~4.5 bits/weight
(4-bit quantization plus one fp16 scale per group of 32 weights):

  3.0e9 x 4.5 / 8 = 1.69 GB

Upper bound on decode rate, ignoring everything else:

  135.2 / 1.69 = ~80 tokens/second

Neither 45 TOPS nor 4.6 TFLOPs appears anywhere in that calculation. That is the point. Below that bandwidth ceiling, adding arithmetic throughput changes nothing, which is why the NPU’s advantage on an interactive chat workload is far smaller than its advantage on a convolutional one — and why weight-only quantization, which reduces the bytes moved, buys more than any engine choice. The same reasoning under a different name is in the KV cache page.

Where the energy actually goes

If the work is dominated by moving bytes, then so is the energy, and this is the part of the argument that is genuine mechanism rather than a product cycle. Mark Horowitz’s ISSCC 2014 keynote, Computing’s Energy Problem (and what we can do about it), published the per-operation energy table that this whole field still cites (IEEE Xplore). At 45 nm, the figures reproduced from it put an 8-bit integer multiply-accumulate on the order of 0.2 pJ and a DRAM access on the order of 640 pJ per 64-bit word — roughly three orders of magnitude apart.

With that ratio in hand, the four real sources of an NPU’s efficiency advantage are all about avoiding movement and overhead rather than about doing arithmetic faster:

• Fewer bits per weight. An engine whose datapath is natively int8 or int4 moves a quarter or an eighth of the bytes an fp32 path would for the same weights. This is the largest single effect and it is a memory effect, not an arithmetic one.
• No per-operation instruction overhead. A GPU fetches, decodes and schedules instructions and reads operands from a register file for every warp. A systolic or dataflow NPU array wires the multiply-accumulates together and streams operands through, so the control energy per MAC approaches zero.
• On-chip reuse. Qualcomm’s HTP keeps working tensors in a dedicated on-chip memory; Intel’s NPU 4 gained access to shared side cache. Every value served from SRAM instead of DRAM is, by the ratio above, one to two orders of magnitude cheaper.
• It runs at a power scale where a GPU is inefficient. A GPU carries fixed costs — clock domains, memory controllers, a driver, a scheduler — that are amortised at high utilisation and dominant at low. An NPU asked to run a small model continuously at a few watts is in its designed operating point; a GPU doing the same is mostly paying to be awake.

And the honest counterpoint, which is the reason this page exists: for a decode-bound LLM, the first mechanism applies and the other three largely do not, because the DRAM traffic is common to both engines. The efficiency gap you can expect on a chat model is real but much smaller than the gap on a vision or audio model, where the arithmetic intensity is high and all four mechanisms are in play.

How to get a real number for your own machine

Nobody publishes joules per token for any combination of model and NPU, and any figure presented as such is somebody’s measurement of one machine. If you need the number, measure it — the method is unglamorous and it is the only thing that answers the question:

1. Fix everything else: same model, same quantization, same prompt, same output length, screen brightness fixed, no background sync.
2. Run a long enough loop that the measurement is not dominated by warm-up — several minutes, not several seconds.
3. Read energy from the platform. On Windows, powercfg /batteryreport gives battery drain over an interval; Intel’s benchmark_app and Qualcomm’s qnn-net-run report execution time on the device, which you combine with a wall-power reading.
4. Divide total energy by tokens generated, and repeat the whole thing with the model on the GPU and on the CPU for the same machine.
5. Report the machine, the driver version and the runtime version alongside the number, because without them it does not transfer to anyone else.

The wider picture of what inference costs in energy terms at scale, rather than on one laptop, is in inference carbon numbers.