Multi-GPU Inference: Tensor and Pipeline Parallelism

Splitting a model across devices is usually described as a way to run bigger models. It is also, in one specific form, the only way to make a single stream generate faster — and in another form, no help to latency at all. Those are different techniques and confusing them is expensive.

Two different reasons to split

• Capacity. The model plus its KV cache exceeds one device. You have no choice; the question is only which split costs least.
• Speed. The model fits, but one device’s bandwidth caps decode at a rate you cannot accept. Splitting the weights across devices means each device reads only its shard, and the bandwidths add.

These two reasons want different strategies, and picking the strategy that matches the wrong reason is the most common way a multi-device deployment disappoints. A configuration chosen for capacity can leave single-stream latency exactly where it was; a configuration chosen for speed can cost more in communication than it recovers in bandwidth. Decide which of the two you are solving before choosing between the strategies below, and write it down, because the diagnostic later depends on knowing what you were trying to buy.

The second reason is the surprising one and it follows directly from the decode bound. With K devices each holding 1/K of the weights:

t_token >= (P * b) / (K * BW_device)  +  K_comm_overhead

Ignoring communication, K devices give a K-times speed-up on
single-stream decode. Communication is what stops it being free.

Tensor parallelism

Each individual matrix is split across devices and every device works on every layer. In a transformer the split follows the structure of the blocks:

• Attention splits by head. Each device owns a subset of the heads and the corresponding slice of the KV cache — which also divides the cache memory, an important secondary benefit.
• The feed-forward block splits column-wise on the first matrix and row-wise on the second, arranged so that exactly one all-reduce at the end reconstitutes the full result.

The result is two collectives per transformer layer — one after attention’s output projection, one after the feed-forward block. Every layer, every token, synchronously.

Two practical constraints come out of the split itself, before any communication cost. The degree must divide the number of key/value heads, or some devices sit idle in the attention block — which is a real limit on models using grouped-query attention with a small number of key/value heads, and it is why a tensor-parallel degree of 8 is sometimes unavailable for a model that happily runs at 4. And the per-device matrices shrink as the degree rises, so at high degrees each device is performing small multiplications that use the hardware less efficiently. Both push toward keeping the degree as low as the memory budget allows.

What tensor parallelism costs per token

The volume is small and the latency is not. For hidden dimension d, bytes per element b, batch B and a ring all-reduce across K devices, each collective moves about:

bytes_per_collective ~= 2 * (K-1)/K * d * b * B

Worked, d = 8192, b = 2, B = 1, K = 4:
    2 * 0.75 * 8192 * 2 = 24,576 bytes  (~24 KB)

Per token, with L = 80 layers and 2 collectives per layer:
    160 collectives, ~3.9 MB total

On a 400 GB/s device-to-device link, 3.9 MB is ~10 microseconds
of transfer. But 160 collectives at a fixed latency of even
5 microseconds each is 800 microseconds of latency.

That is the whole lesson of tensor parallelism in one calculation: at batch size one it is latency-bound, not bandwidth-bound. The bytes are trivial; the number of synchronisation points is not. It is why tensor parallelism works beautifully inside a server with a fast low-latency device-to-device fabric and degrades sharply across a network, and why the degree is normally kept to the devices inside one node.

At large batch the volume term grows linearly in B while the latency term stays fixed, so tensor parallelism becomes bandwidth-limited instead — a regime where a wider link genuinely helps. Which term dominates depends on your batch, and you can tell which by putting your own d, B and link figures into the expression above.

Pipeline parallelism and the bubble

Pipeline parallelism assigns whole layers to devices: device 1 runs layers 1–20, device 2 runs 21–40, and so on. Communication is one activation tensor handed forward per boundary — far less traffic than tensor parallelism, and tolerant of a slower link.

The cost is idleness. With a single request in flight, only one stage is ever working; the rest wait. Splitting the batch into m microbatches keeps more stages busy, but the fill and drain at the ends of the pipeline remain:

bubble_fraction = (S - 1) / (m + S - 1)

S = pipeline stages, m = microbatches

S = 4, m = 1  ->  75% idle
S = 4, m = 8  ->  27% idle
S = 4, m = 32 ->  8.6% idle

So pipeline parallelism buys capacity and throughput when there is plenty of concurrency to slice, and it does nothing whatsoever for single-stream latency — a single token still traverses every stage in sequence, plus the hops between them.

For decode specifically the microbatches come from concurrent requests, so the bubble fraction is a direct function of how busy your endpoint is. A pipeline that looks efficient under load can be 75% idle at three in the morning, and the same configuration therefore has two very different cost profiles depending on traffic. Balancing the stages matters too: layers are not identical in cost, so an even split by layer count is not an even split by time, and the slowest stage sets the rate for the entire pipeline.

Expert and data parallelism

Expert parallelism applies to mixture-of-experts models: different experts live on different devices, and each token is routed to the devices holding the experts it needs. The communication pattern is an all-to-all rather than an all-reduce, and its volume depends on the routing, which depends on the data — so load imbalance is a live problem, and engines add capacity factors and dropping policies to contain it.

Data parallelism is not a split at all: each device holds a complete copy and serves different requests. Zero communication, perfect scaling of throughput, no help for capacity or for single-stream latency. If the model fits on one device and you need more throughput, this is the correct and boring answer, and it should be ruled out before anything more complicated is considered.

Choosing