VRAM arithmetic for local models

Almost every VRAM table on the web is somebody’s recollection. They disagree with each other by a factor of two, they rarely say which model of a given size they mean, and none of them shows the arithmetic — so when your card is 2 GiB short there is nothing to reason with. The numbers are not mysterious. A model’s weights occupy its parameter count times its bits per weight, divided by eight. Its context occupies two tensors per layer, sized by the number of key/value heads and the head dimension, times the context length. The runtime takes a fixed amount before either of those loads. That is the whole model of memory, and every page here is that equation with different numbers in it.

Both inputs are published. Parameter counts fall out of the shapes in a model’s own config; bits per weight is documented per quantization type by the tool that produces the file. So these pages derive rather than assert, and where a figure is an approximation they say which term was approximated. Nothing here was benchmarked — where a page needs a speed, it gives the bandwidth-bound ceiling and the command to measure the real thing on your own machine.