VRAM Requirements: How Much Memory for Which Model

Almost every “will it fit” answer online quotes one term of a four-term sum. The weights are the easy part. At long context and moderate batch, the term people leave out is the one that overflows.

The formula

VRAM = P * b                      weights
     + KV_bytes_per_token * S * B  key/value cache
     + activations(B, S)           transient working set
     + runtime overhead            allocator, kernels, fragmentation

P = parameters
b = bytes per parameter
S = sequence length in tokens (prompt + generated)
B = concurrent sequences (batch)

Every term is arithmetic you can do before you rent anything. None of it requires a benchmark.

Weights, and bytes per parameter

b is set by the numeric format the weights are stored in. Training checkpoints are usually released at bf16 or fp16, two bytes per parameter; quantisation trades precision for bytes.

For a 70B model that is 280 GB, 140 GB, 70 GB and roughly 35–40 GB respectively — before anything else in the sum. Two things follow immediately. Quantisation is the only lever that moves this term at all, and its effect on decode speed is the same factor, because bytes stored and bytes read per token are the same bytes.

For a mixture-of-experts model, use the total parameter count here, not the active count. Every expert has to be resident even though a given token only routes through a few of them. That is why sparse models are cheap per token and expensive to host.

The KV cache, which people forget

Attention needs the keys and values of every earlier token. Recomputing them each step would make generation quadratic, so they are cached — and the cache grows linearly with context and with batch:

KV_bytes_per_token = 2 * L * H_kv * d_head * b_kv

2      = one key and one value
L      = number of layers
H_kv   = key/value heads (with grouped-query attention this is
         far smaller than the number of query heads)
d_head = dimension per head
b_kv   = bytes per cached element

Work it for a model shaped like a typical 70B: 80 layers, 8 key/value heads, head dimension 128, cached at fp16. That is 2 × 80 × 8 × 128 × 2 = 327,680 bytes, about 0.31 MiB per token.

• One sequence at 8k context: 0.31 MiB × 8,192 ≈ 2.6 GiB.
• One sequence at 128k context: ≈ 41 GiB.
• Sixteen concurrent sequences at 8k: ≈ 41 GiB.

At 4-bit weights the same model’s parameters occupy roughly 35 GB. So a single 128k-context request, or a batch of sixteen ordinary ones, needs more memory than the model does. This is the term that turns a configuration that worked in testing into an out-of-memory error in production, because testing used short prompts and one user.

Grouped-query attention is the architectural feature that makes long context affordable at all: it shares key/value heads across query heads, cutting H_kv by the group factor and the cache with it. If a model card quotes query heads and key/value heads separately, that ratio is the number to read.

Activations and runtime overhead

Activations are the transient tensors alive during a forward pass. In decode they are small — one token wide — and a fraction of a gigabyte is a reasonable planning figure. In prefill they scale with the number of tokens processed at once, which is why serving engines offer chunked prefill: it caps the chunk size so a very long prompt does not spike the working set.

Runtime overhead is the part no formula predicts exactly: the CUDA or equivalent context, compiled kernels, communication buffers, and allocator fragmentation. Budget it as a percentage rather than trying to compute it. Paged KV allocation, where the cache is stored in fixed-size blocks rather than one contiguous reservation per sequence, exists specifically to attack the fragmentation part of this term.

Two overheads are large enough to deserve naming separately. Under tensor parallelism, each device holds communication buffers for the collectives that run twice per layer, and their size scales with hidden dimension and batch. And most engines pre-allocate a single large pool at startup rather than allocating per request — which is why a server can report almost all memory in use immediately and still be nearly idle. That reported figure is the pool, not the working set, and reading it as pressure leads people to shrink a batch that was never the problem.

One more thing the formula does not say out loud: splitting a model across devices divides the weights term and the KV term, but it does not divide the per-device overhead. Each device carries its own context, its own kernels and its own buffers. Two 40 GB devices are therefore meaningfully less than one 80 GB device for the purpose of this sum, and the shortfall grows with the number of devices.

A worked budget

A 70B model at 4-bit, serving 8 concurrent sequences at 8k context, on fp16 KV:

weights      70e9 * 0.5 B            =  35.0 GB
KV cache     0.31 MiB * 8192 * 8     =  20.5 GiB  (~22.0 GB)
activations  chunked prefill, say    =   2.0 GB
overhead     ~10% of the above       =   5.9 GB
                                       ---------
total                                 ~ 64.9 GB

Which tells you something a spec sheet does not: this configuration does not fit on one 48 GB device and does fit on one 80 GB device with room to spare — and the thing that decides it is the batch and context you chose, not the model.

Run the same sum with the two variables you control and the shape of your options appears. Raise the context to 32k at the same batch and the KV term alone reaches roughly 82 GB, so the configuration no longer fits anywhere on one device and you are choosing between a smaller batch, an 8-bit cache, or a second device. Drop the batch to two at 8k and the total falls to about 45 GB, which fits the smaller device with headroom. Nothing about the model changed in any of these; the deployment decision was made entirely by the two numbers that get set in a config file and are rarely thought about.

How much headroom to leave

• Size for the longest request you accept, not the average one. The cache is allocated as the sequence grows. If your API allows 128k, some request will use it.
• Decide the batch, do not discover it. Serving engines will happily admit more concurrent sequences until memory runs out. Set the limit from this arithmetic and queue beyond it; queuing is a latency problem, and an out-of-memory abort is an availability one.
• Quantising the KV cache is a separate decision from quantising weights. Most engines let you cache at 8-bit, halving the largest term at long context, and the quality effect is not the same as weight quantisation because it applies only to attention history.
• Leave 10–15% unallocated. Fragmentation and allocator behaviour are real and workload-dependent, and the failure mode of running out is an abort rather than a slowdown.