Choosing a Quantization When You Pull a Model in Ollama

ollama pull qwen3:8b gets you a quantization somebody else chose. The suffix that names it is a convention rather than a standard, and understanding what it encodes turns an eight-way menu into two decisions: how much disk and memory you are prepared to spend, and how much precision you are prepared to lose.

Reading the tag

A library tag is a dash-separated stack of qualifiers after the model name — llama3.1:8b-instruct-q4_K_M is parameter size, then variant, then quantization. The quantization part follows llama.cpp’s naming, and it decomposes:

• The number — nominal bits per weight for the bulk of the tensors. q4, q5, q6, q8.
• The letter — _K marks a k-quant, which groups weights into 256-element super-blocks with their own quantized scales, rather than the older _0 and _1 schemes with 32-element blocks.
• The size letter — _S, _M, _L, meaning small, medium and large mixes. A k-quant does not use one precision throughout: the mix decides which tensors get promoted to a higher precision, which is why a q4_K_M file is larger than four bits per weight would suggest.
• No suffix at all — the default tag. It is an alias, and the manifest reveals what for.

That last point is checkable rather than folklore. The registry manifests for llama3.1:8b and llama3.1:8b-instruct-q4_K_M list the same weights digest and the same size, 4,920,738,944 bytes — so the plain tag is the 4-bit medium k-quant of the instruct model, and pulling both costs one file.

Where the bits per weight come from

The nominal numbers are derivable from ggml’s block structures, published in the llama.cpp source. Each block stores a fixed number of weights in a fixed number of bytes:

q4_0  32 weights in  18 bytes  =  4.50 bits/weight
q8_0  32 weights in  34 bytes  =  8.50 bits/weight
q4_K 256 weights in 144 bytes  =  4.50 bits/weight
q5_K 256 weights in 176 bytes  =  5.50 bits/weight
q6_K 256 weights in 210 bytes  =  6.5625 bits/weight

The extra half-bit over the nominal width is the scale metadata: a block cannot be decoded without knowing what its integers are multiplied by, and that scale is stored alongside. The k-quants get their advantage by amortising that overhead across 256 weights and quantizing the scales themselves, rather than storing a full-precision scale per 32 weights.

A mix such as q4_K_M is then some tensors at q4_K and some promoted to q6_K, which is why its effective rate lands between 4.5 and 6.5625. Nobody publishes the exact promotion table per model, so the honest way to get the real figure is to divide the file by the parameter count. The block sizes above are from ggml’s common header, where each struct carries a static assertion on its own size.

What the registry actually ships

Take the weights-layer sizes Ollama’s registry returns for Qwen3-8B at the time of writing, and Alibaba’s published config.json, from which the parameter count works out at 8.19 billion — 36 layers of about 193 million parameters each, plus untied embedding and output matrices of 151,936 by 4,096:

fp16    16,388,043,552 B  / 8.19e9  = 2.00 bytes = 16.0 bits/weight
q8_0     8,851,075,872 B  / 8.19e9  = 1.08 bytes =  8.6 bits/weight
q4_K_M   5,225,374,496 B  / 8.19e9  = 0.638 bytes =  5.1 bits/weight

The fp16 row is the check on the method: two bytes per parameter is what fp16 means, so the arithmetic is sound. The other two rows land slightly above their block-structure minimums because a GGUF is not uniformly quantized — some tensors are kept at higher precision by the mix, and the file also carries metadata and the tokenizer.

The practical reading: going from fp16 to q8_0 halves the file, and going from q8_0 to q4_K_M takes roughly forty percent more off. Diminishing returns arrive quickly below that — the gap between a 4-bit and a 3-bit build is a few hundred megabytes on an 8B model and a much steeper quality cost.

What you give up

Quantization maps a range of float values onto a small set of integer levels within each block. The error that introduces is small on average and concentrated where it hurts: transformer activations contain outlier features whose magnitudes are far above the rest, and a block containing one of them has its scale stretched, so every other weight in that block is represented more coarsely. This is the mechanism behind quantization damage generally, and it is why the mixed k-quants exist — the promoted tensors are exactly the ones where the outliers live.

Two consequences follow that are easy to miss. Damage is not uniform across tasks: fluent conversational text degrades least, and long chains of exact reasoning, code and rare-language output degrade most, because those are the places where a slightly wrong token choice compounds. And below about four bits the curve steepens sharply, which is why the library stops offering lower builds for most models while the community keeps publishing them. There is no published number that converts a bit width into a quality loss for your task; choosing a quantization treats that side of the decision properly.

Choosing, and changing your mind

The decision is a fit problem before it is a quality problem. Work out what has to be resident — the weights from the table above, plus the KV cache, which is derived from num_ctx and can easily exceed the weights at long context — and pick the largest quantization where the total fits in VRAM with room to spare. A fully resident 4-bit model beats a partially offloaded 8-bit one by a margin that no quality difference recovers, for reasons set out in choosing how many layers to offload.

Changing your mind is cheap in effort and expensive in bandwidth. Pulling a different suffix downloads a whole new weights blob — no delta, no shared layers between quantizations — so both builds sit on disk until you remove the manifest for the one you do not want. Compare them on your own task rather than on a benchmark table, and keep the loser until you are sure. If the quantization you want is not in the library at all, it usually exists as a GGUF elsewhere and can be brought in directly: importing a GGUF file into Ollama is three lines.