llama.cpp's -ngl Flag: How Many Layers to Offload to GPU

Every guide tells you to set -ngl 99 and see what happens. That works right up until it does not fit, and then you need a number. The number is derivable: layer count, weight bytes per layer, KV cache, compute buffer, and whatever your desktop is already using.

What a layer is, and what offloading moves

A GGUF transformer is a stack of near-identical blocks plus a token embedding at the bottom and an output projection at the top. -ngl — spelled --gpu-layers or --n-gpu-layers in full, all three accepted — sets how many of those blocks have their weights placed in device memory instead of system RAM. Everything not offloaded is computed on the CPU, and the activations cross the PCIe bus at the boundary between the two.

That boundary is the whole story. Generation is memory-bandwidth bound: producing one token means reading every weight the token touches, once. A layer resident in VRAM is read at device bandwidth; a layer in system RAM is read at DRAM bandwidth and then its output has to be shipped across the bus. Partial offload is therefore not a smooth ramp — it is two very different speeds averaged by whatever fraction you got onto the card, and the CPU portion dominates the total the moment it is more than a small minority of the stack.

Count the layers before you divide

llama.cpp prints the number you need during model load. Look for n_layer in the llama_model_loader / print_info block:

llm_load_print_meta: n_ctx_train           = 131072
llm_load_print_meta: n_embd                = 4096
llm_load_print_meta: n_layer               = 32
llm_load_print_meta: n_head                = 32
llm_load_print_meta: n_head_kv             = 8

A 32-layer model has 33 offloadable units, not 32: llama.cpp counts the non-repeating output layer separately, which is why -ngl 33 and -ngl 32 behave differently on an 8B and why the log line reads offloaded 33/33 layers to GPU when everything is on the card. If you pass a number larger than the stack it is clamped, and the log tells you what it actually did rather than what you asked for. Read that line every time; it is the only confirmation you get.

Weight bytes per layer are the file size divided by the layer count, near enough. An 8B model at Q4_K_M is about 4.58 GB on disk — that figure is llama.cpp’s own, from the type table in tools/quantize/quantize.cpp, measured against Llama-3-8B — so 4.58 GB over 33 units is roughly 139 MB each. That is the unit you are buying with each increment of -ngl.

The VRAM budget has three lines, not one

The mistake that produces an out-of-memory abort halfway through load is budgeting only for weights. Three things live in device memory:

• Offloaded weights — layers times bytes per layer, as above.
• The KV cache — sized by --ctx-size and, by default, offloaded alongside the weights. llama.cpp prints it as llama_kv_cache: size = ... MiB with the K and V halves itemised. This is the line that surprises people, because it scales with context and not with model size; see what --ctx-size costs in memory for the formula.
• The compute buffer — scratch space for one micro-batch of activations, sized by --ubatch-size rather than by --batch-size. Printed as compute buffer size = ... MiB, and derived in the batch and micro-batch page.

So the derivation is: take free VRAM, subtract the compute buffer, subtract the KV cache for the context you actually want, divide what is left by bytes per layer, and round down. On a 12 GB card with an 8B Q4_K_M and a modest context you will normally find the whole thing fits with room to spare; on a 24 GB card with a 70B at Q4_K_M — about 70e9 × 4.8 bits ÷ 8 ≈ 42 GB of weights before any cache — you will not, and the honest answer is that partial offload of a 70B onto one consumer card is a CPU-speed run with a GPU attached. If you are still choosing the card, VRAM requirements runs the same arithmetic in the other direction.

The default is now auto, and that changes the job

On current master -ngl accepts auto, all, or an integer, and auto is the default. Separately, -fit/--fit defaults to on: llama.cpp estimates the memory the run needs and reduces unset parameters until the estimate fits, leaving a target margin per device of 1024 MiB that you can change with -fitt/--fit-target. It will reduce context down to a floor of 4096 tokens before it gives up.

Two consequences. First, a number you set explicitly is not fitted — setting -ngl 33 and -c 131072 by hand opts you out of the safety net for both, which is exactly what you want when you are benchmarking and exactly what you do not want on a shared machine. Second, if a run that used to work now silently uses less context than you expected, the fitter is the first thing to check: pass -fitp on to have the estimate printed, or use the standalone llama-fit-params binary, which prints the CLI arguments it would have chosen and exits.

Why the last layer is worth more than the first twenty

Partial offload has a cliff at the top, and llama.cpp documents it in its own benchmark example rather than anyone having to assert it. The llama-bench README shows an -ngl sweep over a 7B Q4_0 on CUDA in which generation climbs from 13.45 t/s at -ngl 10 to 40.04 at 30 and 71.76 at 34, and then jumps to 131.66 at 35 — the last layer nearly doubles throughput on its own.

The mechanism is that as long as any layer is on the CPU, every token pays a round trip and a slow read. Once nothing is left on the CPU the round trip disappears entirely and the run becomes purely device-bandwidth bound. This is why “offload most of it” is a much worse position than it sounds, and why the right move when you are two layers short is usually to drop a quantization level or shrink the context rather than to accept 31 out of 33.

You can reproduce the shape of that curve on your own hardware without inventing anything, because llama-bench takes lists and ranges:

1. Build or install llama.cpp with a GPU backend, and confirm the device is visible with llama-bench --list-devices.
2. Read n_layer from the model load output and add one for the output layer.
3. Sweep the top of the range, where the cliff is: llama-bench -m model.gguf -p 512 -n 128 -ngl 24-33. Ranges accept first-last, first-last+step and first-last*mult.
4. Take the highest -ngl that completes without an out-of-memory abort at the context you need — not the one with the best number at the default context, which is a different question.

Mixture-of-experts models want a different flag

For a sparse model, offloading by layer is the wrong cut. Most of the parameters are expert feed-forward weights, only a few of which run per token, while attention runs every time. Putting half the layers on the GPU offloads half of both, when what you want is all of the attention on the device and the idle experts anywhere.

llama.cpp exposes exactly that: -cmoe/--cpu-moe keeps all MoE weights in system RAM, and -ncmoe/--n-cpu-moe N keeps the expert weights of the first N layers there. Combined with -ngl all, this fits a sparse model far larger than your VRAM while keeping the hot path on the card. It is a tensor-level override rather than a layer-level one, which is why it needs its own flag instead of a cleverer -ngl.