Choosing How Many Layers Ollama Offloads to GPU

num_gpu is the number of transformer layers whose weights live in VRAM. Everything else stays in system RAM and is computed by the CPU. That single split explains why a model can be either fast or twenty times slower with no change to the model at all.

What a layer is, in this context

A GGUF model is a stack of repeating transformer blocks plus a token embedding table at the front and an output projection at the back. The offload count refers to the blocks. Ollama’s interactive help describes num_gpu exactly this way — the number of layers to send to the GPU — and it maps directly onto llama.cpp’s -ngl flag, which is where the mechanism actually lives.

A layer is offloaded or it is not; there is no partial layer. During generation the runtime walks the stack once per token, so a model with 36 blocks of which 30 are on the GPU does six CPU round trips per token, each one moving activations across PCIe. This is why the relationship between layer count and speed is not linear: the last few CPU layers cost far more than their share.

What Ollama does if you say nothing

The default value of num_gpu is -1, which does not mean “all layers”. In Ollama’s launcher, -1 means that no -ngl flag is passed at all and the llama.cpp server does its own auto-detection; an explicit 0 passes -ngl 0 and forces CPU-only; any positive number is passed straight through as -ngl N. The comment in the source is unambiguous that -1 is delegation rather than a value.

So on a healthy setup you should not be setting this. The scheduler already accounts for the weights, the KV cache and a headroom allowance before it decides how much fits, and it will spread a model across several GPUs if it does not fit on one. Ollama’s FAQ adds the important preference: if a model fits entirely on a single GPU it is loaded there rather than split, because splitting costs PCIe traffic on every token.

Check the outcome rather than the intention. ollama ps prints a PROCESSOR column reading 100% GPU, 100% CPU, or a split such as 48%/52% CPU/GPU. Anything other than 100% GPU on a machine you believed had room is the signal to investigate, and the usual culprit is the KV cache rather than the weights.

Deriving a layer count from free VRAM

When you do need a number, derive it rather than guess it. The quantity being packed is:

usable = free_vram - kv_cache_bytes - runtime_overhead
layers = floor(usable / bytes_per_layer)

Each term is obtainable. free_vram comes from your driver — nvidia-smi on NVIDIA, and note that it is free memory, not total, since your desktop compositor has already taken some. kv_cache_bytes is the formula from the num_ctx page: two, times layers, times key-value heads, times head dimension, times bytes per element, times context. Runtime overhead is the compute buffers, on the order of several hundred megabytes and worth reserving generously.

bytes_per_layer is the term that needs care. A first approximation is the weights file divided by the block count, but the blocks are not the whole file: the token embedding table is large in models with big vocabularies and is not a repeating block. For Qwen3-8B, whose config published by Alibaba gives 36 layers, a vocabulary of 151,936 and a hidden size of 4,096, the embedding and untied output matrices alone account for about 1.24 billion of the 8.19 billion parameters — roughly fifteen percent of the file that does not divide by 36.

Qwen3-8B, q4_K_M weights file        = 5,225,374,496 bytes
embedding + output share (~15.2%)    ~   794,000,000 bytes
repeating-block share                ~ 4,431,000,000 bytes
divided by 36 blocks                 ~   123,000,000 bytes per layer

That is an estimate with its assumptions stated, not a measurement: it assumes the embedding tensors are quantized at the same average rate as the rest, which they usually are not. Use it to get within one or two layers, then read the exact per-layer buffer sizes out of the server log, which llama.cpp prints on every load. The log is the authority; the arithmetic is for deciding whether to bother trying.

Setting num_gpu

It is an option like any other, so it can be set per request in options.num_gpu, per session with /set parameter num_gpu 28 in the REPL, or per model with PARAMETER num_gpu 28 in a Modelfile. The Modelfile form is the one to be careful with: it travels with the model, so a layer count tuned to your card is baked into anything you share, and it will be wrong on every other machine.

Three cases justify overriding the automatic choice. Running two models at once, where you want to cap the first so the second has room. A machine where the driver misreports free memory, which Ollama’s own GPU documentation notes can happen depending on access rights. And forcing CPU-only with num_gpu 0 to get a baseline, or to keep a background job off a GPU somebody else is using. Reserving VRAM globally is a separate lever — OLLAMA_GPU_OVERHEAD takes a number of bytes to hold back per GPU — and is often the better tool when the goal is to leave room for something that is not Ollama.

Why partial offload hurts more than it looks

Token generation is bandwidth-bound: each token requires reading the active weights once, so the rate is set by how fast those bytes can be read. GPU memory bandwidth is roughly an order of magnitude above typical system memory bandwidth, and the PCIe link between them is slower again. A model split across both runs at the speed of its slowest portion, and the portion in system RAM is doing far more than its share of the waiting.

The practical consequence is that offload is not a smooth dial. Going from all layers on the GPU to a few on the CPU costs disproportionately more than the fraction suggests, and there is usually a better move available than shaving layers: reduce num_ctx so the cache stops crowding the weights, quantize the KV cache with OLLAMA_KV_CACHE_TYPE=q8_0, or drop to a smaller quantization so the whole model fits — see choosing a quantization when you pull. A fully resident smaller model usually beats a partially resident larger one by a wide enough margin to make the quality question secondary.