LM Studio's GPU Offload Slider Explained

The GPU offload control is a wrapper around one number llama.cpp has had for years: how many of the model’s layers to keep in video memory. Knowing that turns the slider from a thing you drag until something works into a calculation you can do first.

What the slider sets

Underneath, LM Studio runs a llama.cpp engine, and this control maps to its layer-offload argument. In llama.cpp’s own server documentation that flag is -ngl, --gpu-layers, --n-gpu-layers N, described as the maximum number of layers to store in VRAM, accepting an exact number, auto, or all, and defaulting to auto.

Two things follow. First, the setting is a count, not a proportion of memory — and layers are not all the same size, so half the layers is not half the memory. Second, the value is clamped: asking for more layers than the model has gives you all of them and no error, which is why all and “some big number” behave identically.

The equivalent on the CLI is lms load, which takes --gpu with max, auto, off or a fraction between 0 and 1, alongside --context-length. That is the form to use in a script, because it does not depend on where a control sits in this release’s layout.

Counting layers and their cost

A transformer’s repeating blocks are its layers, and the count is in the model’s published configuration: Llama 3.1 8B has num_hidden_layers of 32. llama.cpp offloads one more unit than that, because the embedding and output tensors are treated as an extra offloadable unit, so you will see 33 for a 32-layer model. That off-by-one is not a bug and it matters, because the extra unit is often the largest single one — the output projection runs over a 128,256-token vocabulary.

For a first estimate, divide the file size by the unit count. A Q4_K_M build of that model is 4.92 GB per its model card, so:

4.92 GB / 33 units  ~  149 MB per unit (average)

VRAM for weights  ~  units_offloaded x 149 MB
plus KV cache     =  128 KiB per token x context
plus ~0.5-1 GB    =  runtime buffers, compute scratch, desktop

Stated as an average deliberately: the units are not equal, the embedding and output tensors are outliers, and the estimate is good to roughly a unit or two rather than exactly. That is enough to answer the question you have, which is whether you are two layers short or twenty.

Note which term grows when you change your mind. Raising the context does not touch the weight term at all; it grows the cache term linearly, and if the total then exceeds VRAM, the loader compensates by reducing the layer count. That is why raising context sometimes makes generation dramatically slower with no other change — you have silently moved layers back to the CPU. The full cache derivation is on which quant to download.

Setting it and checking it

1. Find your free VRAM with the model unloaded, rather than the card’s nominal size. On NVIDIA that is nvidia-smi; on Apple Silicon, unified memory makes the whole question different and the offload control largely moot.
2. Compute the budget from the block above: weights per unit times the units you want, plus 128 KiB per token of the context you intend to request, plus about a gigabyte of headroom.
3. If the whole model fits, set maximum offload and stop. Everything below this line is only for models that do not fit.

   lms load <model-key> --gpu max --context-length 8192

4. If it does not fit, set the count your arithmetic gave you rather than the maximum, and load. A load that fails outright with a memory error means the estimate was optimistic; drop two units and retry.
5. Check what was actually loaded with lms ps, which reports the models in memory and their footprint. Compare that against your estimate — if it is far off, the average-per-unit assumption was the weak link, and the reported figure is what to plan from next time.
6. Generate something long enough to reach a steady rate, then repeat it at a lower layer count. This is the only reliable way to know whether the last few layers helped on your hardware, and it takes a minute.

Why partial offload falls off a cliff

The intuition people bring is that offloading 90% of layers gives 90% of the speed. It does not, and the reason is worth understanding because it changes what you do about it.

Generating one token requires a pass through every layer in order. Layers on the GPU run at GPU memory bandwidth; layers on the CPU run at system memory bandwidth, which on a consumer machine is roughly an order of magnitude lower. Per-token time is the sum, so the CPU layers dominate as soon as there are any:

time_per_token  =  (gpu_units x t_gpu) + (cpu_units x t_cpu)

with t_cpu ~ 10 x t_gpu, and 33 units total:

  33 on GPU, 0 on CPU  ->  33 t_gpu            (1.0x)
  30 on GPU, 3 on CPU  ->  30 + 30 = 60 t_gpu  (~1.8x slower)
  24 on GPU, 9 on CPU  ->  24 + 90 = 114       (~3.5x slower)

The ratio of ten is an illustrative round number chosen to show the shape, not a measurement of your machine; substitute your own two bandwidth figures and the conclusion does not change. Three units out of thirty-three — under 10% of the model — costs most of the speed, because those three are visited once per token at a tenth of the rate.

The practical rule this produces: full offload, or reconsider. If you are three layers short, the fix is not to accept three layers on the CPU; it is to shrink the context, drop one quant level, or use a smaller model, any of which is likely to buy back the layers and leave you far faster than the partial arrangement. Partial offload is for the case where a model genuinely cannot fit and running it slowly beats not running it.

There is one exception worth naming. In a mixture-of-experts model the experts not routed to for a given token are idle, so keeping some expert weights in system memory costs less than this arithmetic suggests. That is a different calculation and it does not generalise to dense models.

Making the setting stick

A value set while loading applies to that load. LM Studio also keeps per-model defaults — GPU offload, context size and flash attention among them — reachable from the model list, and its documentation states that once set, those settings are used whenever the model is loaded anywhere in the app, including from the CLI.

That last clause is the one to remember, because it becomes confusing later: a server request that just-in-time loads a model picks up the default you saved months ago in the chat interface. If a model behaves differently through the API than in chat, or loads with a context you did not ask for, the saved per-model default is where to look before the request is. Load parameters like these are deliberately not part of a preset — see presets and system prompts for what a preset does carry, and the local server for how a just-in-time load reaches them.