VRAM Requirement Calculator

Parameters, quantisation, context and batch size into the VRAM a model needs — with the weights, the KV cache and the overhead itemised separately.

ParametersIn billions. An 8B model is 8.

Weight precisionBits per parameter: 16 for FP16/BF16, 8 for INT8/FP8, 4 for a 4-bit quant. Bytes per parameter is this ÷ 8.

bit

Layersconfig.json → num_hidden_layers

KV headsnum_key_value_heads. With grouped-query attention this is 4–8× smaller than the attention head count — use the KV one.

Head dimensionhidden_size ÷ num_attention_heads

KV cache precisionBits per element. 16 unless your engine quantises the cache (8 and 4 are common options).

bit

Context lengthTokens resident per sequence, prompt plus generated.

tokens

Concurrent sequences1 for a single local chat; the batch size for a server.

Runtime overheadActivations, workspace, CUDA context and fragmentation, as a share of weights + KV. A placeholder, not a measurement.

VRAM needed

17.81 GB

14.90 GB of weights, 1.00 GB of KV cache for 1 sequence(s) at 8,192 tokens, plus 12% overhead.

Bytes per parameter: 2.00 B
Weights: 14.90 GB
KV cache per token: 128.0 KB
KV cache per sequence: 1.00 GB
KV cache, all sequences: 1.00 GB
Runtime overhead: 1.91 GB
Total: 17.81 GB

What this assumes: GB means GiB (2³⁰ bytes), which is how card capacities are quoted. The KV formula assumes every layer caches keys and values at the same precision — models using multi-head latent attention, sliding windows or cross-layer sharing need less than this says, sometimes far less. Overhead is a percentage you set, not something measured.

Where the number comes from

VRAM = P × (bits/8) + 2 × layers × kv_heads × head_dim × (kv_bits/8) × context × sequences, all × (1 + overhead)

Three terms, and only the first is the one people quote. Weights are fixed the moment you pick a model and a quantisation: parameters times bytes per parameter, so a 4-bit copy of an 8B model is 4 GB of weights and the same model in BF16 is 16 GB. That part is easy, and it is why the “model size” number on a download page feels like the whole answer.

The KV cache is the term that ruins the estimate. Every token you have already processed leaves a key and a value in memory for every layer, and they stay there until the sequence ends. The cost is linear in context and linear in concurrent sequences, so a server holding twenty conversations at 32k tokens can be spending more memory on cache than on the model. The single most consequential field above is KV heads: grouped-query attention shrinks that term by the ratio of attention heads to KV heads, and using the wrong one of those two numbers is how an estimate comes out 8× too large.

The third term is everything the framework needs that is neither weights nor cache — activation buffers for the batch being processed, the CUDA context, allocator fragmentation, and whatever the serving engine reserves up front. It is genuinely hard to predict, which is why it is a percentage here rather than a formula. If you are sizing a purchase rather than an experiment, leave real headroom on top: a model that fits with 200 MB spare does not fit.

VRAM Requirement Calculator

Where the number comes from

The reasoning behind this

Related tools