Local LLMs on a 12GB RTX 3060, by the Numbers

10 min read · updated August 11, 2026

Twelve gigabytes is the size at which the decision stops being “which model” and becomes “which model, at which quant, at which context” — because you cannot have all three.

The two published numbers that matter

NVIDIA’s RTX 3060 product page lists the 12 GB variant as 12 GB of GDDR6 on a 192-bit interface, 170 W graphics card power, and a 550 W recommended system supply (retrieved 11 August 2026). Two consequences follow, and they pull in opposite directions.

The capacity is 12 × 10⁹ bytes = 11.18 GiB, of which roughly 10.6 GiB is usable after the CUDA context and compute buffers. The bandwidth, at the 15 Gbit/s per-pin rate this card’s GDDR6 runs at, is 192 ÷ 8 × 15 = 360 GB/s. That is about a third of a 4090’s, and it means the generation ceiling for a 4.92 GB model is roughly 360 ÷ 4.92 = 73 tokens per second at batch size one, before any runtime overhead. For a card in this class that ceiling is comfortably above what a human reads. Bandwidth is not this card’s problem. Capacity is.

There is also an 8 GB RTX 3060 on a 128-bit bus, which is a different card sold under the same name. Check the memory size and the interface width on the board partner’s sheet before assuming any of the arithmetic below applies.

The 12GB ceiling, derived

Using the same method as the 24GB derivation — weights from published GGUF sizes, KV cache from the model’s own layer and head counts — and assuming 10.6 GiB usable with an fp16 cache:

Llama 3.2 3B Q4_K_M  (2.02 GB)  weights 1.88 GiB  8.7 GiB free  112 KiB/token -> 81k ctx
Llama 3.1 8B Q4_K_M  (4.92 GB)  weights 4.58 GiB  6.0 GiB free  128 KiB/token -> 49k ctx
Llama 3.1 8B Q5_K_M  (5.73 GB)  weights 5.34 GiB  5.3 GiB free  128 KiB/token -> 43k ctx
Llama 3.1 8B Q6_K    (6.60 GB)  weights 6.15 GiB  4.5 GiB free  128 KiB/token -> 36k ctx
Llama 3.1 8B Q8_0    (8.54 GB)  weights 7.95 GiB  2.7 GiB free  128 KiB/token -> 21k ctx

File sizes are the ones published on the bartowski GGUF model cards (retrieved 11 August 2026); the 128 KiB per token comes from 32 layers, 8 KV heads and head dimension 128 at two bytes per element.

Read down that column and the shape of the card becomes obvious. An 8B fits at every quant level from Q4 to Q8 — the choice is not whether it fits but how much context you are trading away for precision. Going from Q4_K_M to Q8_0 costs you 28,000 tokens of context. Whether that is a good trade depends entirely on whether your workload is long-document or short-turn, and nothing about the card tells you which.

A 13B-class model at Q4_K_M lands around 7.5 GiB of weights, leaving about 3 GiB — workable at 16k context, tight at 32k. Anything at or above 20B parameters does not fit at a quant level worth running, and that is where the next section starts.

The partial-offload cliff

Both llama.cpp and Ollama will happily run a model that does not fit, by keeping some layers in VRAM and the rest in system RAM. The layer count is -ngl / --n-gpu-layers. What the tooling does not tell you is how expensive the spill is, and it is expensive for a reason that is pure arithmetic.

Every token read every weight once. Weights in VRAM are read at 360 GB/s. Weights in system RAM are read by the CPU at whatever the host’s memory subsystem gives — a dual-channel DDR4-3200 desktop is 2 × 3200 × 8 ÷ 1000 = 51.2 GB/s of theoretical peak, seven times less. And the two are sequential, not overlapped: the CPU portion of the layer stack must finish before the GPU portion starts on that token.

per-token weight read, 8B Q4_K_M (4.58 GiB), 32 layers

  all 32 layers on GPU   : 4.58 GiB / 360 GB/s   = 13.7 ms  -> ceiling ~73 tok/s
  28 on GPU, 4 on CPU    : 4.01/360 + 0.57/51.2  = 22.3 ms  -> ceiling ~45 tok/s
  16 on GPU, 16 on CPU   : 2.29/360 + 2.29/51.2  = 51.4 ms  -> ceiling ~19 tok/s

Four layers out of thirty-two — twelve per cent of the model — costs about forty per cent of the ceiling. That is the cliff, and it is why “it just about fits with two layers on the CPU” is usually the wrong configuration. Dropping one quant level so the whole thing sits in VRAM will almost always be faster than keeping the higher quant with a spill, and the quality difference between adjacent k-quants is smaller than a forty per cent speed difference is annoying.

One detail makes the spill worse than the arithmetic above suggests. GGUF files are memory-mapped by default, so the CPU-side layers are read through the page cache rather than from a dedicated allocation. On a machine with plenty of free RAM that is fine — the pages stay resident and the read is at memory speed. On a machine where the model plus everything else exceeds physical RAM, those pages get evicted and re-read from storage on subsequent tokens, and the per-token cost stops being 51 GB/s and becomes whatever the SSD does. That is the difference between a slow configuration and an unusable one, and it is the scenario behind most reports of a model generating at under one token per second. Passing --no-mmap forces a real allocation, which fails loudly at load time instead of failing quietly at every token — usually the better outcome.

Configuring for 12GB

Set the context explicitly. Runtimes that default to the model’s full advertised window will try to allocate a cache for 131,072 tokens, which is 16 GiB on an 8B and instantly fatal on this card. -c 8192 in llama.cpp; in Ollama the equivalent is the num_ctx parameter on the model or the request.
Quantize the cache before you quantize the weights. --cache-type-k q8_0 --cache-type-v q8_0 halves the cache, which at 32k context on an 8B recovers 2 GiB — more than a full quant step down would.
Run the display off the integrated GPU if you have one. A desktop session on the 3060 can hold half a gigabyte or more, which at this size is several thousand tokens of context.
Do not leave two models loaded. Ollama keeps a model resident after a request; the keep-alive duration is configurable and the default has changed between releases. Check what yours is before concluding the card is smaller than it is.

Finding your own layer count

The arithmetic gives a ceiling. The number that matters is the largest -ngl that loads without an out-of-memory error at your context length, and the fastest way to find it is to sweep, which llama-bench does natively:

Pick your real context length and fix it: -c 8192, not the default.
Sweep the layer count — llama-bench -m model.gguf -ngl 24,28,30,32 -n 128 -r 3 — and read the tokens-per-second column.
If the top of the sweep errors out with CUDA error: out of memory, that is the answer: see the OOM page for what to reduce first.
Re-run the winning configuration with --cache-type-k q8_0 --cache-type-v q8_0 and see whether a full offload is now possible at a quant level you preferred.