Running an LLM on Your Own Laptop: A Complete Guide

Whether a model fits on your machine is not a matter of opinion or of finding the right forum post. It is four numbers and a multiplication, and the numbers are all printed on the model’s config file.

The memory arithmetic

Three things occupy memory when a model is loaded: the weights, the key-value cache, and everything else. Only the first two scale with anything you control.

Weights

The weights are the parameter count times the bits each parameter is stored in, divided by eight:

weight_bytes = params * bits_per_weight / 8

# bits_per_weight, roughly, for common formats:
#   fp16 / bf16   16.0
#   int8 / Q8_0    8.5
#   Q6_K           6.6
#   Q5_K_M         5.7
#   Q4_K_M         4.8     <- the usual default
#   Q3_K_M         3.9
#   Q2_K           3.35

The quantised figures are above their nominal bit width because K-quant formats store per-block scales alongside the packed weights, and because some tensors — embeddings, attention output — are commonly kept at higher precision than the name suggests. Treat them as close enough for planning; the file size shown on the download page is the authority, and it is the number you should actually subtract from your budget.

The KV cache

Every token you have processed leaves a key and a value vector in every layer, and they stay resident for the life of the conversation. This is the part people forget, and it is why a model that loaded fine runs out of memory twenty minutes into a long session.

kv_bytes = 2 * layers * kv_heads * head_dim * ctx * elem_bytes * batch

# 2         one key and one value
# layers    num_hidden_layers
# kv_heads  num_key_value_heads  (NOT num_attention_heads, under GQA)
# head_dim  hidden_size / num_attention_heads
# ctx       tokens currently held
# elem_bytes 2 for fp16, 1 for an 8-bit KV cache
# batch     concurrent sequences

Every field there is a key in config.json in the model repository. The distinction that matters is num_key_value_heads versus num_attention_heads: grouped-query attention shares key and value heads across query heads, and a model with 32 query heads and 8 KV heads has a cache four times smaller than the naive calculation. Nearly every recent model does this.

Everything else

Activation and compute buffers, the CUDA or Metal context, and the fragmentation you will not predict. Reserve a gigabyte or so on a dedicated card, more on a laptop where the display is also drawing from the same pool.

A worked example

Take an 8B-parameter dense model with 32 layers, 32 attention heads, 8 KV heads and a hidden size of 4096, which describes a great many of the models in that class. Head dimension is 4096 / 32 = 128.

weights at Q4_K_M
  8e9 * 4.8 / 8 = 4.8e9 bytes ~= 4.5 GiB

KV cache, per token, fp16
  2 * 32 * 8 * 128 * 2 = 131,072 bytes = 128 KiB

KV at 8k context, one sequence
  128 KiB * 8192 = 1.0 GiB

KV at 32k context, one sequence
  128 KiB * 32768 = 4.0 GiB

total, 8k context   4.5 + 1.0 + ~1.0 overhead = ~6.5 GiB
total, 32k context  4.5 + 4.0 + ~1.0 overhead = ~9.5 GiB

Two lessons fall straight out. Context is not free — going from 8k to 32k cost more than the difference between two quantisation levels would have. And if you are short, quantising the KV cache to 8 bits halves that term, which is often a better trade than dropping the weights another level.

Reading the answer back as a tier

Run the same calculation for the size classes and you get the tiers people usually quote, except now you can see where they came from and adjust them when your context is unusual.

• Around 8 GB of usable memory. Models up to roughly 8B at a four-bit quant, at moderate context. Comfortable for classification, extraction, summarising and rewriting.
• Around 16 GB. The same models at higher precision and much longer context, or a mid-size model at four-bit. This is the first tier where a coding assistant is pleasant rather than a demonstration.
• Around 24 GB. Mid-size dense models at four-bit with long context, or a small model plus an embedding model plus headroom for concurrency.
• Unified-memory machines. Apple Silicon and similar architectures let the GPU address most of system memory, which changes the arithmetic entirely: capacity stops being the binding constraint and memory bandwidth becomes it. Large models load and then generate slowly.
• Mixture-of-experts models. Every expert must be resident even though only a few run per token. Budget on total parameters, expect throughput closer to the active count.

Getting one running

Two commands and you have a local endpoint. With Ollama, which manages downloads and keeps a server running:

ollama run llama3.1:8b-instruct-q4_K_M

# see what is loaded and how it was split between GPU and CPU
ollama ps

# raise the context window for a session
OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve

With llama.cpp directly, which gives you every knob and an OpenAI-compatible server:

llama-server \
  -m ./models/model-Q4_K_M.gguf \
  -c 8192 \           # context; this is the ctx in the KV formula
  -ngl 99 \           # layers offloaded to GPU; 99 means "all of them"
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --host 127.0.0.1 --port 8080

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"hello"}]}'

The flag to understand is -ngl. If the whole model does not fit on the GPU, layers stay on the CPU and every token pays a round trip across the bus for them. Partial offload is not a graceful degradation; it is a cliff. If your calculation says it does not fit, drop a quant level or shorten the context rather than accepting a split.

Measuring your own token rate

Published throughput numbers are close to useless to you, because they are a statement about somebody else’s memory bandwidth, thermal headroom and driver version. Generate your own in about a minute:

# prompt processing (prefill) and generation, reported separately
llama-bench -m ./models/model-Q4_K_M.gguf -p 512 -n 128

# how the rate changes as the cache fills
llama-bench -m ./models/model-Q4_K_M.gguf -p 4096 -n 256

Read the two columns separately. Prompt processing is compute-bound and usually fast; generation is bound by how quickly the machine can stream the weights out of memory, which is why the quantisation level affects generation speed almost as much as it affects capacity. Run the second invocation too: generation slows as context grows, and a rate quoted at an empty cache flatters every setup.

When there is no GPU

CPU-only inference works and is worth knowing about. Generation speed tracks system memory bandwidth, so it is roughly an order of magnitude below a discrete card, and prompt processing suffers more than generation does. That makes it unsuitable for chat with long prompts and entirely suitable for batch work that runs overnight, for models in the 1–4B class doing narrow tasks, and for anything where the alternative is not doing it at all. Set -ngl 0, use a four-bit quant, keep the context tight, and judge it on completed jobs per hour rather than on tokens per second.