bitsandbytes NF4 Quantization Explained

Int4 gives you sixteen values evenly spaced across a range. Weights are not evenly spread across their range — they cluster hard around zero — so most of those sixteen slots land where almost no weights are. NF4 moves the slots to where the weights actually are, and the sixteen constants it moves them to are published.

What uniform int4 wastes

Take a weight tensor and normalise it so its largest absolute value is 1. Uniform symmetric int4 then represents values at sixteen evenly spaced points across [-1, 1], roughly one every 0.133. The weights themselves are approximately normally distributed with a standard deviation far below that maximum — for a tensor whose extreme value sits around four standard deviations out, about 68% of the weights fall inside [-0.25, 0.25], which is four of the sixteen slots. The other twelve slots share the remaining third of the weights, and the outermost ones may be occupied by a handful of values each.

That is the waste. Quantization error is what the reader cares about, and error is dominated by the dense region where most of the weights live. Spending three quarters of your codebook on the sparse tails buys almost nothing.

Quantile quantization, and its one problem

The information-theoretically optimal placement for a fixed number of levels on a known distribution is at its quantiles: choose the bin edges so that every bin holds an equal share of the probability mass. Each of the sixteen codes then describes the same number of weights, and no code is wasted on a region that is nearly empty.

The problem with quantile quantization in general is that estimating quantiles from data is expensive and, for the tails, badly conditioned — the extreme quantiles are exactly the ones you have the fewest samples for. The QLoRA paper (Dettmers, Pagnoni, Holtzman and Zettlemoyer, 2023) sidesteps this with an assumption it can defend: neural network weights are close enough to zero-centred normal that you can compute the quantiles of a standard normal once, offline, in closed form, and then rescale each block of real weights into that distribution rather than estimating anything at runtime.

That is what NormalFloat is. A fixed codebook derived from the normal distribution’s quantiles, plus a per-block scale that maps real weights onto it.

The sixteen values

The codebook is not a mystery: it is sixteen float constants in bitsandbytes, generated by create_normal_map in functional.py. Written out, in order from code 0 to code 15:

-1.0
-0.6961928009986877
-0.5250730514526367
-0.39491748809814453
-0.28444138169288635
-0.18477343022823334
-0.09105003625154495
 0.0
 0.07958029955625534
 0.16093020141124725
 0.24611230194568634
 0.33791524171829224
 0.44070982933044434
 0.5626170039176941
 0.7229568362236023
 1.0

Three properties of that list are worth naming because each was a design decision.

• The spacing is tight in the middle and loose at the edges. The gap between the two smallest positive values is about 0.081; the gap between the two largest is about 0.277. That ratio is the whole point — resolution is spent where the density is.
• Zero is exactly representable. Code 7 is 0.0, not an approximation of it. A format without an exact zero cannot represent a padded or masked weight without introducing error, and an asymmetric quantile split is what buys the exact zero here.
• The endpoints are exactly ±1. The block’s largest absolute weight maps to a code that reproduces it exactly after rescaling, so the extreme value in every block survives quantization intact. Because a true quantile split of a symmetric distribution into 16 bins with an exact zero would need an odd count, the construction uses an asymmetric split — eight negative levels and seven positive, plus zero — which is why the list above is not symmetric about zero.

Blocks, absmax and double quantization

The codebook lives on [-1, 1], so each block of real weights is divided by its own absolute maximum before being matched to a code and multiplied by it again on the way back out. In bitsandbytes that block is 64 weights, which puts NF4 at the fine end of the granularity spectrum — see quantization granularity.

Fine blocks cost storage, and here is the arithmetic. One FP32 absmax per 64 weights is 32 bits over 64 weights, or 0.5 bits per weight on top of the 4 — a 12.5% overhead, which is a lot to pay for a scale. Double quantization is the fix: the absmax values are themselves quantized, to 8 bits, in blocks of 256, with an FP32 scale over each of those. The overhead becomes:

per weight = 8 bits / 64        (quantized absmax)
           + 32 bits / (64*256)  (scale over the absmaxes)
           = 0.125 + 0.00195
           = ~0.127 bits per weight

so NF4 with double quantization ≈ 4.127 bits per weight
   NF4 without                   ≈ 4.5   bits per weight

For a 7B model that difference is about 0.33 GB, which is exactly the margin that decides whether something fits. It is off by default and costs one flag:

from transformers import BitsAndBytesConfig
import torch

cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # default is "fp4"
    bnb_4bit_use_double_quant=True,     # default is False
    bnb_4bit_compute_dtype=torch.bfloat16,
)

Where NF4 is and is not the right format

NF4 quantizes on load, from the full-precision checkpoint, with no calibration data and no measurement pass. That is its defining practical property and it cuts both ways.

• It needs nothing from you. No calibration corpus, no hours of GPU time, no separate artefact to distribute. Any checkpoint on disk becomes a 4-bit model in the time it takes to read it. Formats that calibrate — GPTQ, AWQ — have a bake step that has to happen once per model and produces a file somebody must host.
• It is the format QLoRA fine-tuning is built on. The frozen base stays in NF4 while LoRA adapters train in higher precision on top, which is what made single-GPU fine-tuning of large models practical. If you are training rather than only serving, this is usually the reason NF4 is in the stack.
• It gives up the kernel advantage. Because no calibration reshaped the weights and no packing convention is shared with the GPTQ/AWQ ecosystem, NF4 inference runs through bitsandbytes’ own dequantize-and-matmul path rather than the heavily tuned mixed-precision kernels described in Marlin. For serving throughput that difference is the one that shows up.

The short version: NF4 is the format for getting a model into memory quickly and for training under it. A calibrated format is the one for serving it to other people.