vLLM’s “No Available Memory for the Cache Blocks” Error

vLLM loads the weights, profiles a forward pass, then tries to carve what is left into KV blocks. When there is nothing left it refuses to start, and it tells you to raise the setting that is often the reason you have the problem.

The exact string

ValueError: No available memory for the cache blocks. Try increasing
`gpu_memory_utilization` when initializing the engine.

It is raised during engine initialisation, before the server binds a port, so nothing has served a request yet. The message and its history are visible in the vLLM tracker — issue 2248 is the long-running thread on it, and issue 5274 is the one that captures the trap: a high value gives an out-of-memory crash, a low value gives this error, and the window between them can be narrow.

How vLLM decides the cache size

The startup sequence is worth knowing in order, because each step is a different way to run out.

• Weights load. Fixed cost, computable in advance: parameters times bytes per parameter, plus a little for buffers.
• A profiling run executes. vLLM runs a dummy forward pass at the maximum batch and sequence length it has been configured for, and records peak memory. This is the activation cost, and it scales with --max-num-batched-tokens.
• CUDA graphs are captured, unless --enforce-eager is set. Graph capture holds its own memory.
• Whatever remains becomes KV blocks. The budget is gpu_memory_utilization × total minus everything above. If that is zero or negative, you get this error.

The load-bearing detail is in the fourth line and it is the reason the same flag behaves differently on two machines: gpu_memory_utilization is a fraction of the device’s total memory, not of its free memory. Set 0.9 on an idle 80 GB card and vLLM plans for 72 GB. Set 0.9 on the same card with 30 GB already held by another process and vLLM still plans for 72 GB, of which only 50 exists — and it will discover that either here or, worse, as a crash later.

Four causes, and how to tell them apart

Run nvidia-smi before you change any flag. The answer to which of these you have is usually visible in its output.

• The fraction is genuinely too low for the model. Free memory in nvidia-smi is large, nothing else is on the card, and you passed something conservative like 0.5. This is the case the message’s advice is written for. Raise it.
• Another process is on the GPU. nvidia-smi lists a second PID — a notebook kernel, a previous vLLM that did not exit, a second container. Because the fraction is of total memory, vLLM has already over-promised. Kill the other process; do not raise the fraction, which will convert this into an out-of-memory crash.
• The weights barely fit at all. Weight bytes are close to the whole card. A 70B at fp16 is roughly 70e9 × 2 = 140 GB before anything else, so it does not fit on one 80 GB device at any utilisation setting; that is a tensor-parallel or a quantization decision, not a flag.
• Fragmentation on a large multi-layer model. Contributors on issue 5274 describe free memory falling steadily through the decoder layers of a 70B, leaving a small remainder even when the arithmetic said there should be room. This one is the hardest to distinguish and --enforce-eager is the cheapest thing to try against it.

The error it is often confused with

vLLM has a second startup failure that reads similarly and has the opposite fix. It appears when the KV cache was allocated but is too small to hold one sequence of the requested length — the message names the model’s maximum sequence length and the number of tokens the cache can store, and points at --max-model-len. If you see that one, lowering --max-model-len fixes it. If you see the cache-blocks error, the allocator got nothing at all and shortening the context will not conjure memory that another process is holding.

Fixing it without trading one OOM for another

1. Clear the card first. nvidia-smi, then terminate anything that is not this server. A crashed previous run frequently holds its allocation.
2. Raise the fraction in small steps — 0.85, then 0.90 — and treat 0.95 as the ceiling. The remainder is not waste; it absorbs allocator fragmentation and the CUDA context.
3. Add --enforce-eager. It skips CUDA graph capture, which frees the memory those graphs hold at the cost of some per-step overhead. If this alone makes the server start, you now know the graphs were the marginal cost.
4. Reduce --max-num-batched-tokens or --max-num-seqs. Both shrink the profiling peak, which is subtracted before the KV budget is computed.
5. If none of that leaves room, the model is too large for the device. Raise --tensor-parallel-size across more GPUs, or serve a quantized checkpoint — what quantization costs at inference is the trade you are making.

There is one more lever that people reach for last and should reach for earlier, which is the width of the cache itself. --kv-cache-dtype fp8 stores keys and values at eight bits instead of sixteen, which roughly halves the per-token cost of the KV cache and therefore roughly doubles the number of blocks the same remainder affords. It is a quality trade and it is not appropriate for every model, but it is the only option on this list that increases KV capacity rather than merely finding memory somewhere else. The general shape of that trade is in the KV cache page.

Do the sums before any of this if you can. Weight bytes are parameters times bytes per parameter: a 7B at fp16 is about 7e9 × 2 = 14 GB, at fp8 about 7 GB, at 4-bit about 3.5 GB plus quantization metadata. Subtract that from gpu_memory_utilization × total, subtract a couple of gigabytes for activations and CUDA graphs, and what is left is your real KV budget. If that number is negative on paper it will be negative at runtime, and no combination of flags will change it — you are looking at a different device, more devices, or a smaller checkpoint.

A last note on ordering: run the two commands below in the order shown, because starting a second server before the first has released the device is the single most common way to arrive back at this error having changed a flag and concluded, wrongly, that the flag did nothing.

nvidia-smi --query-compute-apps=pid,used_memory --format=csv
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --enforce-eager