Running Two Consumer GPUs Without NVLink for Local Inference

Two 24 GB cards give you 48 GB of usable weight space and no NVLink bridge to join them. For single-stream generation that turns out to matter far less than the training literature suggests, and the reason is a single number: how many bytes cross the bus per token.

Two ways to split a model, and they differ

There are two distinct ways to put one model on two cards, they have completely different communication patterns, and almost all confusion about multi-GPU inference comes from treating them as one thing.

Layer split — also called pipeline parallelism — puts the first half of the layers on card 0 and the second half on card 1. A token’s activations pass through the first stack, cross to the second card once, and pass through the second stack. This is llama.cpp’s default, exposed as --split-mode layer.

Tensor split — tensor parallelism — cuts every individual weight matrix in half and puts half on each card, so both cards work on every layer simultaneously and must combine their partial results before the layer can finish. llama.cpp calls this --split-mode row; vLLM calls it --tensor-parallel-size 2.

Layer split is latency-neutral and bandwidth-cheap. Tensor split can be faster because both cards compute at once, but it pays for that with a synchronisation on every layer. Which one you can afford is entirely a function of the link between the cards.

Layer split: what actually crosses the bus

Under a layer split, the only thing that moves between the cards during generation is the hidden state at the boundary: one vector of hidden_size values per token. For Llama 3.3 70B, the published config gives a hidden size of 8,192 across 80 layers. At two bytes per element that is 16 KiB per token, once.

hidden state crossing the boundary, 70B, fp16
  8192 values * 2 bytes = 16,384 B = 16 KiB per token, one crossing

time over PCIe 4.0 x16 (31.5 GB/s):  16,384 / 31.5e9  = 0.52 us
time over PCIe 4.0 x4  ( 7.88 GB/s): 16,384 /  7.88e9 = 2.08 us

Two microseconds. A token on a 70B model takes tens of milliseconds to produce even on fast hardware, so the transfer is on the order of one ten-thousandth of the step. This is the whole argument: under a layer split, the interconnect is not in the critical path in any meaningful way, and NVLink’s absence costs you approximately nothing.

What the interconnect does cost you is load time, because all 42.5 GB of a Q4_K_M 70B has to reach the cards through it once. That arithmetic, including what a narrow slot does to it, is on the PCIe lanes page.

The other cost is that layer split does not make generation faster. Card 0 works while card 1 waits, then card 1 works while card 0 waits. You get capacity, not speed — the two cards together produce tokens at roughly the rate one card would if the model fitted on it. The gain is that the model runs at all, at full VRAM residency, instead of spilling to system RAM.

Mismatched cards work under a layer split, and this is one of its better properties: there is no requirement that the two halves be equal, only that each card holds the layers assigned to it. A 24 GB card paired with a 12 GB one gives 36 GB of weight space, provided the split is weighted so the smaller card takes proportionally fewer layers. What you do not get is the faster card’s speed — the token step is the sum of both stacks, so the slower card’s share of the layers costs a proportional share of the time, and the pair generates at something between the two cards’ individual rates rather than at the better one.

The KV cache follows the layers rather than living on one card, which matters when you are budgeting: each card holds the cache for the layers it owns, so a long context is divided between them in the same ratio as the weights. That is convenient, and it is also why an uneven split that just fits at 4k context can fail at 32k — the card with the tighter margin runs out first, and the error names that device rather than the split.

Tensor split: where the interconnect bites

Tensor parallelism inverts the picture. Each layer’s output must be combined across both cards before the next layer starts, which means an all-reduce of a hidden-state-sized tensor per layer, in both directions, at every layer boundary. For an 80-layer model that is on the order of 160 crossings per token instead of one.

80 layers * 2 crossings * 16 KiB = 2.5 MiB per token

over PCIe 4.0 x16 (31.5 GB/s): 2.62e6 / 31.5e9 = 83 us  per token
over PCIe 4.0 x4  ( 7.88 GB/s): 2.62e6 /  7.88e9 = 333 us per token

Still small in absolute terms, but now it is a real fraction of a step rather than a rounding error, and it is latency-dominated rather than bandwidth-dominated: 160 small synchronised transfers per token pay 160 round-trip latencies, and PCIe round trips are microseconds each where NVLink’s are not. This is the regime NVLink was built for, and it is why datacentre parts have it and consumer parts do not.

The practical conclusion for two consumer cards without a bridge: use a layer split for single-stream generation. Tensor split earns its communication cost when you are serving many concurrent requests, where both cards being busy on the same layer is worth the synchronisation — the same argument that makes batching pay.

Peer-to-peer, and what consumer cards will not do

There is a second constraint that surprises people, and it is a driver policy rather than a physical limit. CUDA peer-to-peer — one GPU reading another’s memory directly across PCIe, without staging through host RAM — is not enabled on recent GeForce cards. cudaDeviceCanAccessPeer returns false, and frameworks that assume peer access is available either fall back to a host-staged copy or fail outright.

The symptom is usually not an error message about peer access. It is vLLM or a similar server hanging at startup during its NCCL initialisation, or logging that it is disabling a peer-to-peer path. The fallback path — device to host to device — roughly doubles the transfer cost and adds a synchronisation, which under a layer split is still irrelevant and under a tensor split is not.

llama.cpp does not depend on peer access for its default layer split, which is one reason it is the pragmatic choice for a two-consumer-card machine. If you are setting NCCL_P2P_DISABLE=1 to get a server to start, that is the constraint you have met.

Setting it up

1. Confirm both cards are visible and note their order: nvidia-smi --query-gpu=index,name,memory.total --format=csv. Device indices are assigned by the driver and are not necessarily slot order.
2. Run a layer split explicitly rather than relying on a default that may change: llama-server -m model.gguf -ngl 99 --split-mode layer -c 8192.
3. If the cards have different capacities, weight the split so neither runs out: --tensor-split 24,12 assigns layers in that ratio. Without it the runtime divides evenly and the smaller card fails first.
4. Watch both cards’ memory during load with nvidia-smi --query-gpu=index,memory.used --format=csv --loop-ms=500. A split that is silently putting everything on one card shows up immediately.
5. Compare the split against a single-card baseline with llama-bench before concluding it helped. For a model that already fits on one card, a split is usually slower.