Skip to content
Multigrid

KV Cache Memory Simulator: Users × Context → GiB

Exact KV cache arithmetic from your model's shape, with a concurrency slider against a memory bar that shows the moment the card runs out.

GPU memory
■ weights 16.00 GiB■ overhead 8.00 GiB■ KV cache 15.63 GiB□ free 40.38 GiB
KV cache for 16 concurrent users
15.63 GiB

Fits, with 40.38 GiB to spare. This card tops out at about 57 concurrent full-length sequences before the cache alone exhausts it.

KV bytes per token, per sequence
131,072 B
…as a formula
2 × layers × kv heads × head dim × bytes
One full 8,000-token sequence
1000.0 MiB
× 16 users
15.63 GiB
Model weights
16.00 GiB
Runtime overhead
8.00 GiB
Left for KV cache
56.00 GiB
Maximum concurrent sequences
57
What this assumes: every sequence is assumed to hold its full context at once, which is the peak case and the one that decides whether you fall over — average occupancy is lower and a paged cache reclaims freed blocks, so real steady-state use sits below this. The formula counts keys and values separately (the leading 2) and uses KV heads, not attention heads: under grouped-query attention those differ by the group factor and using the wrong one inflates the answer several times over. Sizes are binary GiB. Runtime overhead is a percentage you supply, not a measured figure — profile your own server if the answer is close.

The memory nobody budgets for

Model weights are the number everyone knows and the KV cache is the number that decides capacity. Weights are fixed: load them once and they occupy the same bytes whether you serve one request or a hundred. The cache is per-sequence and per-token, so it scales with exactly the two quantities your product grows on — how many people are using it and how much context each of them has.

Drag the user count and the shape of the problem is immediate. A model whose weights comfortably fit becomes a model that cannot serve twenty people at long context, and no amount of weight quantisation fixes that, because you were never short of weight memory. What fixes it is fewer KV bytes per token: grouped-query attention, which is why the KV-head field is separate from the head count; an 8-bit KV cache, which halves the whole term; or simply shorter contexts, which is the lever people reach for last and which works linearly.

The per-token figure is worth internalising on its own. At 32 layers, 8 KV heads, head dimension 128 and two bytes, one token costs 128 KiB of cache — so a single 8,000-token conversation is a gigabyte. That is the sentence that explains why long-context serving is expensive in a way that per-token pricing does not: the provider is not just doing more arithmetic, they are holding a gigabyte of your conversation resident on a card that costs more per hour than most instances cost per day.

Prefix caching changes the economics but not this arithmetic — a shared prefix is stored once instead of per user, which is a large win precisely because the per-user number here is so large.

The reasoning behind this

Related tools

KV Cache Memory Simulator: Users × Context → GiB · Multigrid