Setting Up a Home Lab for Local LLMs

Every home lab question — which card, how much RAM, does a Mac count — reduces to two numbers: how many bytes the weights occupy, and how many bytes the KV cache occupies at the context you intend to use. Work those out first and the hardware tier picks itself.

The arithmetic that decides everything

Weights. Parameter count times measured bits per weight, divided by eight. For a 4-bit k-quant that is roughly 4.9 bits per weight rather than 4, for reasons worked through in why file size never matches parameter count. An 8B at Q4_K_M is about 4.9 GB; a 70B at Q4_K_M is about 43 GB, both figures published in llama.cpp’s llama-quantize README.

KV cache. This is the term buying guides skip, and it is what actually stops a model fitting. Per token, the cache stores a key and a value for every layer:

bytes per token
  = 2 (K and V)
  x n_layers
  x n_kv_heads
  x head_dim
  x bytes_per_element

Llama-3.1-8B, fp16 cache, from its published config:
  32 layers, 8 KV heads, head_dim 128, 2 bytes
  = 2 x 32 x 8 x 128 x 2
  = 131,072 bytes = 128 KiB per token

  at   8,192 tokens ->  1.0 GiB
  at  32,768 tokens ->  4.0 GiB
  at 131,072 tokens -> 16.0 GiB

So the same model is a 6 GB problem at 8k context and a 21 GB problem at full context. That single fact reorders most hardware advice: a 12 GB card runs an 8B model perfectly well until somebody asks it to read a long document. The mechanism behind the growth is in the KV cache explainer; the lever is context length, and secondarily cache quantization, which runtimes expose as an 8-bit or 4-bit KV type and which halves or quarters the term above at some quality cost.

Add a gigabyte or so for compute buffers and the runtime itself, and you have your requirement. Do this before reading any tier below.

The tiers, and what each one buys

Tier 0 — the machine you already own

CPU inference with system RAM. This is not a consolation prize: a 3B or 4B model at Q4 needs under 2.5 GB and will run on anything from the last decade. It is bandwidth-bound, so it is slow, and the ceiling is derivable rather than guessable — dual-channel DDR4-3200 is 2 × 8 bytes × 3200 MT/s = 51.2 GB/s theoretical, and generating one token requires reading every weight once, so 51.2 ÷ 1.9 GB gives an upper bound near 27 tokens per second for a 3B at Q4 before any inefficiency. Real throughput is a fraction of that ceiling. Spend nothing here until you know what you actually want to run.

Tier 1 — one 8–12 GB GPU

The first tier where generation feels immediate. An 8B at Q4 fits with room for 8k–16k of context by the arithmetic above. A 12 GB card gives you the same model at a longer context, or a 14B at Q4 with a short one. This is the tier most people should stop at, because the jump in usefulness from tier 0 is large and the jump from here is not.

Tier 2 — one 24–32 GB GPU

NVIDIA publishes 24 GB on the GeForce RTX 3090 and 4090 and 32 GB on the RTX 5090. This tier runs a 24B–32B model at Q4 with a working context, or an 8B at Q8 with a very long one. What it does not do is run a 70B at Q4: 43 GB of weights does not fit in 32 GB, and no amount of context trimming changes that. Offloading the remainder to system RAM works and is slow, because every token now waits on the slowest path in the machine.

Tier 3 — two 24 GB cards, or 64–128 GB of unified memory

48 GB is the first configuration that holds a 70B at Q4 with a modest context. On Apple silicon, unified memory is usable as model memory, and Apple’s Mac Studio technical specifications list 410 GB/s of memory bandwidth for the base M4 Max, 546 GB/s for the higher configuration and 819 GB/s for M3 Ultra. Those are published figures for the memory system, not throughput measurements; running models on Apple silicon goes into what that buys in practice.

Tier 4 — very large unified memory

Hundreds of gigabytes of unified memory will hold models nothing else in a house will hold, particularly sparse mixture-of-experts models where total size is large and per-token work is small. The catch is asymmetric: generation is bandwidth-bound and these machines have bandwidth, but prefill is compute-bound and a discrete GPU has far more arithmetic throughput. A long prompt can therefore feel slow on a machine that generates quickly — the two phases are limited by different things, which is the mechanism described in memory bandwidth and inference speed.

Why two cards are not twice as fast

Two 24 GB cards give you 48 GB of capacity. They do not give you double the tokens per second, and expecting otherwise is the most common disappointment at tier 3.

The usual split for a single stream is by layer: the first half of the model on one card, the second half on the other. Generating a token means passing through layer 1, then 2, and so on, so the cards work one after the other rather than at the same time. Each card is idle while the other computes, and there is a transfer across the bus at the boundary. Capacity adds; single-stream latency does not improve and can get slightly worse. Where multiple cards do pay off is throughput across concurrent requests, because then there is work for both at once — the same reason GPU utilisation looks so poor on a single interactive session.

The parts nobody budgets for

• Power and the circuit. Two high-end cards plus a CPU under load is a serious sustained draw, and a home lab that trips a breaker under load has a hardware problem that looks like a software one. Check the power supply’s rating against the sum of the cards’ published board power, not against their idle.
• Physical space and airflow. Two triple-slot cards do not fit in most consumer cases with usable spacing, and a card starved of airflow throttles.
• Idle draw. A machine that is on all day so the model is warm costs money all day. This is the largest hidden line in the local versus API cost comparison.
• Storage. Model files are large and you will collect them. Keep them on an SSD: loading 40 GB of weights from a spinning disk on every cold start is a minutes-long wait every time the server restarts.
• Noise. Server cards are passively cooled and expect a chassis with loud fans. This is a genuine reason people abandon a home lab.

What to buy in what order

1. Run the model you think you want on hardware you do not own — rented, borrowed, or a smaller quantization on your existing machine — and confirm it is actually good enough for your task. Model quality at a given size is the variable that decides whether the hardware was worth buying, and it is covered in how good local models actually are.
2. Compute the requirement: weights plus KV cache at the context you need, plus a gigabyte of headroom. Write the number down.
3. Buy the smallest single card that holds that number. One card that fits beats two that need splitting.
4. Add RAM before adding a second GPU. System RAM is cheap and lets you hold a larger model slowly, which is often enough to find out whether you want to hold it quickly.
5. Only then consider a second card, and only if your workload has concurrency for it.