A Local LLM Setup for Under $500: What the Budget Actually Buys

A budget build is one decision with several consequences. Almost every number that decides whether a model will run at all is a memory number, and the arithmetic that connects a parameter count to a gigabyte figure is simple enough to do before you open a single listing.

You are buying VRAM, and very little else

Token generation on a local model is bounded by memory, in two separate senses. Capacity decides whether the model fits; if it does not, layers spill to system RAM and every token pays a PCIe round trip, which is a different order of slowness rather than a proportional one. Bandwidth then decides the ceiling on speed, because producing one token requires reading essentially every weight the model uses for that token. Neither of those is improved by a faster CPU, more system RAM, or a better cooler.

So the budget question reduces to: how many gigabytes of fast memory can this money buy, and how fast is that memory. Everything else in the build exists to keep the card fed and powered. That framing is also why comparing cards on gaming benchmarks is misleading here — a card that wins on rasterisation and has less memory is the worse buy, sometimes by a category.

From parameter count to gigabytes

Start with the only two inputs you need: how many parameters, and how many bits each one is stored in. A quantised GGUF stores most weights at a target bit width with a small per-block scale, so the effective rate is not a round number. Take Q4_K_M at approximately 4.8 bits per weight, which is the figure the llama.cpp quantisation tables quote, and the conversion is:

bytes = parameters x bits_per_weight / 8

 8.0e9 x 4.8 / 8 =  4.8e9  bytes  ~=  4.8 GB
14.0e9 x 4.8 / 8 =  8.4e9  bytes  ~=  8.4 GB
32.0e9 x 4.8 / 8 = 19.2e9  bytes  ~= 19.2 GB
70.0e9 x 4.8 / 8 = 42.0e9  bytes  ~= 42.0 GB

Those are the results of that arithmetic under that assumption, not measurements. The honest way to check one is to read the file size off the model card on Hugging Face before downloading, because the publisher lists it per quantisation and that number is the ground truth for the specific conversion you are about to use. Note that some weights are gated and require accepting the licence with the publisher before the files are downloadable at all — Meta’s Llama releases work this way, and the Llama 3.1 8B Instruct card on Hugging Face shows both the access gate and the architecture constants used below.

Bits per weight is the one lever that changes this figure without changing the model, and it changes it a lot: the same 32B model is 19.2 GB at 4.8 bits and 32 GB at 8 bits. Which level is defensible for which job is the subject of choosing a quantisation level by use case, and the variant letters are covered in the Q4_K variants.

The second number nobody budgets for

Weights are the number people quote. The KV cache is the number that decides whether the configuration you actually wanted will load. Every token in the context stores one key and one value vector per layer, and those stay resident for the whole generation. The size is:

bytes_per_token = 2 (K and V)
                x layers
                x kv_heads
                x head_dim
                x bytes_per_element

Llama 3.1 8B: 32 layers, 8 KV heads (grouped-query),
head_dim 128, FP16 cache (2 bytes):

  2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB per token

  8,192-token context ->  1.0 GiB
 32,768-token context ->  4.0 GiB
131,072-token context -> 16.0 GiB

The layer count, KV-head count and head dimension are published in the model’s config on its Hugging Face card; the arithmetic above is those constants multiplied out. The lesson is in the last line: on this architecture a full 128k context costs more than three times what the 4-bit weights cost. Grouped-query attention is what keeps it merely expensive — with 32 KV heads instead of 8 the same figure would be four times larger, which is why older architectures became impractical at long context.

Add a compute buffer on top for the activations of the current batch, which scales with the micro-batch size (-ub, documented default 512) and the context. A working rule for planning is weights plus KV plus roughly a gigabyte, and then leave headroom, because a model that fits with 100 MB spare will fail on a longer prompt rather than at load time. The trade between context length and quantisation level is worked in detail in the context-and-quant tradeoff.

What each VRAM tier unlocks

Applying both calculations, with an 8k context and a gigabyte of overhead:

• 8 GB. An 8B at Q4 (4.8 + 1.0 + ~1 = ~6.8 GB) fits, with little room to grow the context. A 14B does not. This tier works and constrains you to one model class.
• 12 GB. The same 8B with a 32k context (4.8 + 4.0 + ~1 = ~9.8 GB), or a 14B at Q4 with a short context. This is the first tier where you have a choice to make rather than one option.
• 16 GB. A 14B at Q4 with a useful context, or a 14B at Q6 for a quality-sensitive job. Still short of 32B.
• 24 GB. A 32B at Q4 with 8k of context (19.2 + ~1.5 + ~1 = ~21.7 GB) — tight, and the reason 24 GB cards are discussed as the threshold for that model class.
• Unified memory. Apple Silicon shares one pool between CPU and GPU, so the capacity question is answered by system RAM rather than by a separate VRAM figure. The bandwidth of that pool is published per chip and is the number that then decides speed.

Below the bottom tier, CPU-only inference is a real option with real and quite different expectations — what to expect without a GPU is worth reading before concluding that a budget build requires a card at all.

How to price this today, honestly

No price appears on this page, and that is deliberate. Used graphics card prices are not published by anyone whose figure could be cited and dated; they move week to week with supply, and a stale number in a page like this is worse than no number, because a reader will spend against it. What is durable is the method.

1. Fix the target from the arithmetic above. Decide the model class and context length first, add the weights, KV and overhead, and write down the VRAM figure. That is your requirement; everything after it is shopping.

2. Rank candidate cards on gigabytes per unit of money, then on bandwidth. Capacity is a hard gate and bandwidth is a speed multiplier, so they are not interchangeable. Both are published by the manufacturer for every card; take them from the vendor specification page rather than from a listing.

3. Check the architecture, not just the memory. Very cheap high-VRAM cards are usually cheap because they are old. Datacentre cards from before the tensor-core era have no accelerated FP16 path and no support for the fused attention kernels modern runtimes assume, so they can hold a large model and still run it slowly; some also have no display output and passive cooling that a desktop case does not provide.

4. Price the rest of the machine. The PSU needs the right connector count and headroom above the card’s rated board power; the case needs the physical length; the disk needs to hold several quantisations of several models, which adds up faster than people expect — budgeting disk for local models covers that separately.

5. Record the date next to the number. Whatever total you arrive at is a statement about one week. Write the date on it so that the next person reading your notes knows what to re-check.