The Interconnect: Why Communication Is the Hidden Scaling Limit

A model split across devices spends part of every layer waiting for the other devices. Whether that wait is negligible or dominant is decided by two properties of the link, and for interactive inference it is usually the one people ignore.

The hierarchy, and its orders of magnitude

Every level of the memory and communication hierarchy is roughly an order of magnitude slower and higher-latency than the one above it. The exact figures change every generation; the ordering does not, and the ordering is what you design against.

Two numbers describe a link

The standard model for the time to move a message is linear in its size with a fixed offset:

t(n) = alpha + n / beta

alpha = latency: fixed cost per message, in microseconds
beta  = bandwidth: bytes per second once flowing
n     = message size in bytes

The crossover size n* = alpha × beta is where the two terms are equal. Below it you are paying for latency; above it, for bandwidth. For a link with 5 microseconds of latency and 400 GB/s, that crossover is 5e-6 × 4e11 = 2 MB. Messages smaller than about two megabytes are latency-dominated on that link, and their transfer time is essentially independent of how large they are.

That last sentence is the one people find surprising and it is worth dwelling on. Below the crossover, sending twice as much data costs almost nothing extra — the message was going to cost alpha regardless. So the instinct to reduce communication volume, which is correct for large transfers, is close to useless for small ones. The productive move for small messages is to send fewer of them: fusing several collectives into one, or overlapping them with computation so the latency is hidden rather than reduced. Serving engines that overlap communication with the next layer’s computation are exploiting exactly this.

The floor on per-token time

Now connect it to inference. Tensor parallelism performs two collectives per transformer layer, and the message in each is one activation tensor: hidden dimension times batch times bytes per element. At d = 8192, bf16, batch 1, that is 16 KB — four orders of magnitude below the 2 MB crossover, so every one of those collectives costs essentially alpha and nothing else.

t_comm_per_token  >=  2 * L * alpha_collective

L = 80 layers, alpha_collective = 5 us
    -> 160 * 5 us = 800 us = 0.8 ms per token

which alone caps decode at ~1,250 tokens/second,
before any weight is read.

Two consequences fall out of that arithmetic. First, adding devices past the point where the weight-reading term has shrunk below the communication term makes single-stream decode slower, not faster — the classic scaling curve that turns over. Second, buying a fatter link does not help this case at all: you are paying alpha, and alpha is set by protocol overhead, switch hops and software launch cost rather than by width.

At large batch the message size grows past the crossover and the situation reverses: bandwidth binds, and a wider fabric is exactly the right purchase. Same hardware, opposite advice, and the batch size is what selects between them.

Work out where your own deployment sits and the purchase decision becomes concrete. The message size is hidden dimension times batch times bytes per element; set it equal to alpha × beta for your link and solve for the batch. With d = 8192 at bf16, a message reaches 2 MB at a batch of about 128. Below that concurrency you are buying latency and a wider link changes nothing; above it you are buying bandwidth and it changes a great deal. This is also why the same cluster can show excellent scaling on a throughput benchmark and disappointing scaling on an interactive one, with no configuration difference between them at all.

Topology and collective algorithms

The time a collective takes also depends on how it is performed. A ring all-reduce moves an optimal amount of data — about 2(K−1)/K times the tensor size — but takes 2(K−1) steps, so its latency grows linearly with the number of participants. A tree reduction takes O(log K) steps and moves more data. Libraries switch between algorithms based on message size for exactly the reason the alpha-beta model predicts: small messages want few steps, large messages want minimal volume.

Topology decides how many hops a step takes. All-to-all connectivity within a server means one hop between any pair; a switched fabric adds switch latency; crossing a node boundary adds a network stack. Each hop is added to alpha, and alpha is multiplied by 160 collectives per token.

Some of alpha is not the wire at all. Launching a collective costs work on the host and on the device, the participants must synchronise before data moves, and a network path adds protocol processing on both ends. This is why techniques that bypass the operating system and let devices read each other’s memory directly exist, and why they matter more for inference than the raw bandwidth number they are usually advertised with: they attack the fixed cost, and the fixed cost is what a decode step pays over and over.

What this changes in practice

• Keep tensor parallelism inside a node. This is the most repeated piece of advice in distributed inference and the alpha-beta model is why: crossing a node boundary multiplies alpha, and alpha is paid 160 times per token.
• Use the slow link for the sparse traffic. Pipeline parallelism sends one tensor per stage boundary rather than two per layer, which is why it is the strategy that tolerates a network.
• Distrust scaling curves without a stated batch size. The interconnect binds through latency at batch one and through bandwidth at large batch. A curve measured in one regime says nothing about the other.
• Watch for stragglers. A collective completes when its slowest participant arrives, so one thermally throttled device sets the pace for every device in the group — which is where cooling stops being a facilities problem and becomes a throughput one.