Why GPUs and Not CPUs

“GPUs are parallel” is the usual answer and it is only half right. Parallelism is why a GPU can do the arithmetic at all; bandwidth is why it finishes. For the part of inference you wait on most, the second one is the whole story.

What inference asks the hardware to do

A forward pass through a transformer is a chain of matrix multiplications with a few cheap element-wise operations between them. Every output number is a dot product of a row of activations with a column of weights, and no output depends on any other output in the same multiplication. There is no branching to predict and nothing to serialise.

That shape splits the work into two regimes, and they stress different parts of the machine. Prefill reads your whole prompt at once, so every one of its tokens goes through the model together and each weight loaded from memory gets reused across all of them. Decode produces one token per pass, so a weight is loaded, used once, and discarded. Same arithmetic, opposite bottleneck.

Parallelism: one instruction, a lot of data

A CPU core is built to make one instruction stream finish quickly: deep caches, aggressive branch prediction, out-of-order execution, wide single-thread issue. Those are all expensive per-core, so you get a few dozen of them. A GPU spends the same silicon on thousands of simple lanes running the same instruction over different data, plus matrix units that do a small matrix multiply as a single operation. When the work is a million independent dot products, the second layout wins by a margin no clock speed closes.

The execution model is worth naming because it explains the failure cases as well as the successes. Lanes are grouped and issued together, so all the lanes in a group execute the same instruction on different data. When a program branches and different lanes take different paths, the hardware executes both paths and masks off the lanes that should not participate — divergence costs you real throughput. A matrix multiply never diverges, which is why it maps onto this hardware as well as any workload ever has, and why a workload full of data-dependent branching maps onto it badly.

This is why the arithmetic throughput gap is large, and it is also why the gap does not translate into a proportional speed-up for token-by-token generation. You cannot use arithmetic units you cannot feed. A useful way to hold the two facts together: parallelism decides the ceiling on how much arithmetic the device could ever perform, and bandwidth decides how much of that ceiling any particular operation is allowed to reach.

Bandwidth: the number that binds

To generate one token, the runtime must read every weight the token needs. For a dense model that is all of them. So the minimum bytes moved per token is:

bytes_per_token = P * b          (weights)
                + KV bytes read  (grows with context)

P = parameter count
b = bytes per parameter (2 for fp16/bf16, 1 for fp8, 0.5 for 4-bit)

And the hard ceiling on single-stream decode follows immediately, because you cannot produce a token faster than you can read the bytes it needs:

tokens_per_second  <=  memory_bandwidth / bytes_per_token

Nothing about the model architecture or the kernel quality can beat that bound. Better software gets you closer to it; nothing gets you past it.

It is worth being precise about which memory this refers to. The weights live in the device’s own memory, and the bandwidth in the formula is the bandwidth between that memory and the compute chip — not the link to the host, and not the on-chip cache bandwidth, which is far higher but far too small to hold a model. Accelerators reach their bandwidth figures by stacking DRAM dies directly beside the logic and connecting them with a very wide interface; a conventional system moves data over a comparatively narrow bus to modules some distance away. The architectural difference is physical, and it is the reason the gap between the two categories has stayed roughly an order of magnitude across generations rather than closing.

Putting numbers on it

Take a 13-billion-parameter dense model at bf16. Weights are 13e9 × 2 = 26 GB, so every decoded token requires at least 26 GB of reads plus a smaller KV term. Now compare two devices by their memory bandwidth alone:

Notice what happened to the model. Divide any two of those results and the 26 GB cancels: for single-stream decode of the same model, the speed ratio between two devices is their bandwidth ratio. Peak FLOPs never entered the calculation. That is the sharpest form of the argument for accelerators, and it is also the warning that a device with impressive arithmetic and ordinary memory will disappoint you at batch size one.

Prefill inverts it. With S prompt tokens sharing one weight read, the arithmetic per byte loaded rises roughly in proportion to S, so a prompt of a few hundred tokens moves the work into the region where peak FLOPs matter and the matrix units are the thing you are buying. Both regimes are real; they simply bill you for different parts of the chip.

Where CPUs still win

• Small models at low volume. A 1–3B model quantised to 4-bit reads a couple of gigabytes per token. On a machine with 100 GB/s of system memory the bound is tens of tokens per second, which is faster than a person reads.
• Capacity you already own. System RAM is cheap and plentiful. A model that does not fit in accelerator memory and would otherwise need multiple devices may run acceptably on a CPU with enough RAM, slowly but without a second machine.
• Everything around the model. Tokenisation, retrieval, reranking with small models, request handling, JSON validation. Sending those to an accelerator adds transfer latency to work the CPU finishes in microseconds.
• Idle economics. An accelerator you rent by the hour costs the same whether or not a request arrives. A CPU you already have costs nothing extra to leave idle, which matters far more for bursty traffic than peak throughput does.

The honest summary: GPUs win because transformer inference is a wall of independent multiply-accumulates attached to a very large array of weights, and accelerators are the only widely available devices that are wide enough and fed fast enough to serve both halves of that description.