Consumer GPUs for Local LLMs: What Fits in Each VRAM Tier

“What can I run on this card” has two halves, and only one of them requires owning the card. What fits is arithmetic. How fast it goes is bandwidth divided by bytes. Both are on the box.

Capacity is arithmetic, speed is bandwidth

Rearrange the memory formula to solve for the model size a tier can hold:

P_max = (VRAM - KV - activations - overhead) / b

at 4-bit,  b ~= 0.55 bytes/parameter in practice
           (0.5 nominal, plus per-group scales and
            some tensors kept at higher precision)

And the speed that follows, once it fits:

tokens_per_second  <=  BW / (P * b)

Consumer graphics memory bandwidth has spanned roughly the high hundreds of GB/s per card in recent generations, an order of magnitude rather than a specification. Put your card’s actual figure in. The useful consequence is that within a generation, two cards with the same memory type and bus width generate at similar speeds for the same model — the difference between them shows up in prefill, which is compute-bound, and in whether the model fits at all.

That consequence is worth stating as a buying rule because it contradicts the way cards are marketed. Within a generation, the tier above usually brings more memory, a wider bus and more arithmetic together, and the reviews foreground the arithmetic because games respond to it. For local inference the ordering is the reverse: the memory decides which models are available to you at all, the bus width decides how fast they generate, and the arithmetic decides only how quickly long prompts are read. A card that wins a gaming comparison can lose a local-inference comparison to a slower card with more memory, and the arithmetic above is what shows why.

Tier by tier

Derived from the formula above, assuming 4-bit weights, a modest 8k context for a single user, and about 1.5 GB reserved for the operating system, the display and runtime overhead. These are capacity results, not test reports.

The quantisation format deserves a sentence of its own, because “4-bit” names a family rather than a scheme. The practically important variable is the group size: weights are quantised in blocks with a shared scale, and a smaller block preserves more information while costing more bytes for the scales. That is why the effective bytes per parameter sits above 0.5 and why two 4-bit builds of the same model can differ in both size and quality. Some formats additionally keep the most sensitive tensors — often the embedding and output layers — at higher precision, which is a disproportionate share of the size in a small model.

Two cards of the same tier can also be combined, at which point you are doing multi-GPU inference over whatever link the consumer platform gives you, and the interconnect term stops being negligible. On a consumer platform that link is usually the general-purpose expansion bus rather than a dedicated device-to-device fabric, so tensor parallelism — which synchronises twice per layer — performs far worse than the same configuration would in a server. Splitting by layer instead, so each card owns a contiguous block and hands one activation tensor forward, is the arrangement that suits the hardware, and it buys capacity rather than speed.

The context tax nobody budgets for

Every tier above assumed 8k of context for one user. The KV cache is linear in context, so the model you can hold shrinks as you extend it. For a 13B-shaped model — 40 layers, 8 key/value heads, head dimension 128, fp16 cache — the per-token cost is 2 × 40 × 8 × 128 × 2 = 163,840 bytes, about 0.16 MiB.

• 8k context: ≈ 1.3 GiB. A tolerable slice of a 12 GB card.
• 32k context: ≈ 5.1 GiB. Nearly half of it.
• 128k context: ≈ 20 GiB. More than the card.

This is why a local setup that answers short questions instantly falls over on a long document, and why 8-bit KV caching is often the single most valuable setting in a local runtime: it halves the term that grows.

Offloading, and why it hurts so much

Every local runtime will let you keep some layers in VRAM and the rest in system RAM. The arithmetic explains the result better than any review can. If a fraction f of the weights lives in system memory, the time per token is bounded by the sum of two reads:

t_token >= (1-f) * P * b / BW_vram  +  f * P * b / BW_host

BW_host here is the effective path to system memory, which is
bounded by the PCIe link as well as by DRAM.

Because BW_host is typically an order of magnitude below BW_vram, even a small f dominates the sum. Offloading 20% of a model does not cost you 20% of your speed; the offloaded fraction contributes proportionally to its much slower term. Fitting entirely in VRAM, even at a lower precision, is almost always faster than a higher precision that spills.

Choosing on your own workload

Everything above is capacity and bandwidth arithmetic, which is the part that can be settled without owning anything. What it cannot settle is whether a model of the size that fits is good enough for what you want, and that question has no general answer — it depends on the task, and a 14B model that handles your classification perfectly may be useless for the reasoning you also wanted. So treat the tier table as a filter that tells you which models are candidates, and treat quality as a separate test you run on your own prompts.

• Decide the model class first, then buy the tier that holds it with context. Working backwards from a card leads to quantising past the point where quality holds.
• Prefill speed matters more than you expect for coding. Pasting a file is a long prompt, and long prompts are compute-bound, so this is the one case where a card’s arithmetic throughput genuinely shows.
• Quantisation damage is task-dependent. A 4-bit model that answers general questions well can degrade noticeably on exact-format output or multi-step arithmetic. Test on your own task before treating a tier as sufficient.
• Local is a privacy and control decision more than a cost one. The hardware is a fixed cost paid whether or not you generate a token, which makes it excellent for continuous use and expensive for occasional use — the same crossover that shows up in renting versus buying.