Serving a Model With vLLM

vLLM turns a checkpoint into an OpenAI-compatible HTTP server with one command. What it is really giving you is two things — continuous batching and a paged key-value cache — and every flag worth knowing about is a knob on one of those. Understanding the pair means the configuration stops being trial and error.

What vLLM actually does

Two mechanisms account for the throughput difference against a naive serving loop.

• Continuous batching. A naive server groups requests into a batch and waits for the slowest to finish. vLLM adds and removes sequences from the running batch at every decoding step, so a request that arrives mid-flight joins the next step and a request that finishes frees its slot immediately. On mixed traffic this is the larger of the two wins.
• Paged key-value cache. The cache is allocated in fixed-size blocks rather than as one contiguous reservation per sequence, the same idea as virtual memory paging. That removes the need to reserve worst-case context for every concurrent request, which is what lets many more sequences share a GPU, and it makes shared prefixes genuinely shareable across requests.

The consequence for configuration: nearly everything you tune is about how much memory the cache gets and how long a sequence is allowed to be. Those two numbers, together, set your concurrency.

One command, one endpoint

# Start a server. The subcommand and these flags have been stable
# for a long time; check --help on your version before adding others.
vllm serve <model-id> \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --tensor-parallel-size 1

# It speaks the OpenAI chat API:
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "<model-id>",
       "messages": [{"role": "user", "content": "hello"}],
       "max_tokens": 64}'

The OpenAI compatibility is the practical headline. Any client library pointed at this base URL works unchanged, which means swapping between a hosted provider and your own server is a configuration change — the same property that makes a local Ollama server usable from application code written against a hosted API. The server also applies the model’s chat template for you on the chat endpoint, which removes an entire class of formatting bug.

The flags that matter

The arithmetic behind the flags

This is what turns the flags from guesses into decisions. Every value comes from the model’s config file or from your own choice.

Step 1 — what the weights take
  weights_GB = parameters × bytes_per_parameter / 1e9
  7B at bfloat16 = 14.0 GB

Step 2 — what is left for the cache
  cache_GB = total_GPU_GB × gpu_memory_utilization − weights_GB
  on an 80 GB card at 0.90:  80 × 0.90 − 14.0 = 58.0 GB

Step 3 — cache cost of one token
  bytes_per_token = 2 × layers × kv_heads × head_dim × bytes_per_element
  32 layers, 8 kv heads, head_dim 128, bfloat16:
  2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KB

Step 4 — how many sequences fit
  tokens_that_fit = cache_GB × 1e9 / bytes_per_token
                  = 58.0e9 / 131,072 ≈ 442,000 tokens

  at --max-model-len 8192, worst case:  442,000 / 8,192 ≈ 54 sequences
  at --max-model-len 32768, worst case: 442,000 / 32,768 ≈ 13 sequences

The final two lines are the whole point. Quadrupling the maximum length quarters your worst-case concurrency, on the same hardware, with no other change. If your application sends 3,000-token prompts and asks for 500-token answers, setting the limit to the model’s full advertised context is throwing away three quarters of your capacity to serve requests you will never receive.

Two honest caveats. Paged allocation means the worst case is only reached when sequences actually grow to the limit, so real concurrency is usually better than the number above — that is precisely the point of paging. And architectures differ: multi-head-latent attention and similar designs change the per-token formula substantially, so take the shape from the config rather than from this page.

The three startup failures

Almost every failed start is one of these, and they are distinguishable from the message.

1. Out of memory while allocating the cache. The weights loaded and there was not enough left. Fixes, in order of preference: lower --max-model-len; use a quantised checkpoint; raise the utilisation fraction if it is conservative; add a GPU and raise tensor parallelism. Raising utilisation past about 0.95 usually trades a startup failure for a mid-traffic one.
2. A shape or divisibility error with tensor parallelism. The head count is not divisible by the number of GPUs. This is a property of the model, not a bug; use a value that divides.
3. The requested context exceeds what the model supports. Asking for more than the checkpoint’s trained maximum is refused rather than silently allowed, which is the correct behaviour. Override mechanisms exist for models with extendable context and should be used knowingly, because quality beyond the trained length degrades in ways benchmarks on shorter inputs will not show — the topic of effective context length.

A fourth failure appears in production rather than at startup: requests being queued rather than served, because the cache is full. The server exposes metrics for running and waiting sequences and for cache utilisation; those are the numbers to alert on, not GPU utilisation, which will look busy either way.

Prefix caching, and prompt order

Because the cache is paged, blocks holding an identical prefix can be shared between requests instead of recomputed. That is the second large win after continuous batching, and unlike the flags above it is something your application controls rather than the server.

The rule that falls out of it: put the stable part of the prompt first. A long system prompt, a fixed set of tool definitions and a shared set of few-shot examples are identical across every request, so if they are at the front they are computed once and reused. Put a timestamp, a user name or a request id before them and every request has a unique prefix from token one, which throws the entire saving away for the sake of a field nobody reads.

Shared prefix, reused across requests:
  [ system prompt ][ tool definitions ][ few-shot examples ][ user turn ]
   ^--------------- identical, cached ----------------^   ^ unique ^

Unique from the first token, nothing reusable:
  [ "Request 8fa2 at 14:03" ][ system prompt ][ tools ][ user turn ]
   ^ unique ^

The effect is largest exactly where prompts are longest — retrieval systems with a big instruction block, agents carrying many tool definitions — and it reduces prefill work, which is the part of the request that dominates time to first token. The same ordering discipline applies to hosted providers with prompt caching, and it is set out in context and cache ordering.

Measuring your own throughput

Published throughput figures are not transferable — they depend on the model, the hardware, the prompt length distribution and the concurrency. The project ships benchmarking scripts, and the honest approach is to run one against your own traffic shape rather than quoting anyone’s numbers, including this page’s.

1. Take your real length distribution. Median and 95th percentile of prompt tokens and of output tokens, from your logs. Two numbers each. Benchmarks run at fixed lengths mislead in proportion to how far they sit from these.
2. Sweep concurrency, not batch size. Send 1, 4, 16, 64 concurrent requests at those lengths and record total tokens per second and per-request latency at each. The interesting output is the point where latency starts climbing faster than throughput.
3. Report percentiles. A mean hides the queued requests, which are the ones users complain about — the argument is in latency percentiles.
4. Re-run after every flag change. Especially after changing the maximum length or the utilisation fraction, because both change concurrency rather than per-request speed, and only a concurrency sweep shows that.

For how vLLM compares with other engines on the same hardware, the existing comparison in serving engines covers the trade-offs; this page is about getting one running well.