Can I Run This Model?

Enter your card's memory and bandwidth and a model's shape, and get a fit verdict, the largest context that fits, and the memory-bandwidth ceiling on tokens per second.

Your VRAMPut your own card's figure here — or your unified memory budget on Apple silicon.

Your memory bandwidthFrom your card's own specification. The example is a round placeholder, not any particular device.

GB/s

Bandwidth efficiencyShare of peak bandwidth your runtime actually achieves. Set it from your own benchmark; the default is a guess and it scales the answer linearly.

ParametersIn billions.

Weight precisionBits per parameter. For a GGUF file you already have: file bytes × 8 ÷ parameters.

bit

Context lengthTokens you want resident, prompt plus generation.

tokens

Layersconfig.json → num_hidden_layers

KV headsnum_key_value_heads. With grouped-query attention this is 4–8× smaller than the attention head count — use the KV one.

Head dimensionhidden_size ÷ num_attention_heads

KV cache precisionBits per element. 16 unless your engine quantises the cache (8 and 4 are common options).

bit

Runtime overheadShare of weights + KV, for activations and allocator slack.

Verdict

Fits

18.71 GB spare. Ceiling of about 124.2 tokens/s for one stream at this context — an upper bound from bandwidth alone, not a prediction.

Weights: 3.73 GB
KV cache at this context: 1.00 GB
Runtime overhead: 0.57 GB
Total required: 5.29 GB
Your VRAM: 24.00 GB
Headroom: 18.71 GB
Largest context that fits: 145,025 tokens
Bytes read per decode step: 4.73 GB
Effective bandwidth: 630 GB/s
Single-stream ceiling: 124.2 tok/s
Time per token at the ceiling: 8 ms

What this assumes: The token rate is a ceiling derived from memory bandwidth: one decode step must read every weight and this sequence’s KV cache, so it cannot be faster than those bytes divided by bandwidth. Real rates land below it — sampling, attention kernels, CPU overhead and thermal limits all subtract. Prefill is not included; it is compute-bound and is a separate wait before the first token. The KV term uses your full stated context, so early in a generation the rate is higher than shown.

Why bandwidth, not TFLOPs, decides this

tokens/s ≤ (bandwidth × efficiency) ÷ (weight bytes + KV bytes so far)

Generating one token means running the whole model once. Every weight has to travel from memory into the compute units, and at batch size one each of those weights is used for a single multiply before it is discarded. That makes single-stream decoding a memory-bandwidth problem, not an arithmetic one — the card’s TFLOPs figure barely enters into it. Divide bandwidth by the bytes that must move per step and you have the fastest the hardware can possibly go.

The consequence is worth internalising: quantising a model to half the bits roughly doubles the ceiling, because the same computation moves half the bytes. It is the same reason a model that spills a few gigabytes into system RAM does not get slightly slower but catastrophically slower — the spilled portion now moves at PCIe or host memory speed, which is an order of magnitude down.

The fit half of the verdict is stricter than it looks. Fitting means fitting the weights, the KV cache for the context you actually intend to use, and the runtime’s working set, all at once. Plenty of setups load a model successfully and then fail thousands of tokens into a conversation, because the cache grows with every token and nothing warned anyone. If you want a specific context length, size for it up front rather than discovering the limit in production.

Can I Run This Model?

Why bandwidth, not TFLOPs, decides this

The reasoning behind this

Related tools