Serving Engines Compared: The Features That Decide Throughput

Published throughput comparisons between serving engines go out of date within a release or two, and half of them are confounded by a configuration difference the author did not control. The mechanisms they are really comparing change much more slowly, and once you know them you can read any engine’s release notes and predict what it will do to your workload.

Why this compares mechanisms, not numbers

Nothing here was benchmarked by us, and the honest reason to avoid quoting somebody else’s numbers is that a throughput figure is meaningless without the model, the precision, the input and output length distribution, the concurrency, the hardware, and the version of every component. Change any one and the ranking can invert.

So this page does two things instead: it explains the five mechanisms that account for most of the difference between engines, with the arithmetic showing why each matters, and then gives a protocol for measuring them on your own workload — which is the only comparison that answers your question anyway.

Continuous batching

With static batching, a fixed group of sequences runs to completion together. Sequences finish at wildly different lengths, so slots go idle while the longest one continues. If lengths vary as they do in real traffic, average occupancy of the batch over its lifetime can be roughly half.

Continuous batching — sometimes called iteration-level scheduling — re-forms the batch at every decode step, admitting a queued request into any slot that has freed. The arithmetic benefit is direct: below the ridge point, throughput is proportional to the batch actually running, so recovering idle slots recovers throughput almost one-for-one. This is the single largest structural difference between a modern engine and a naive server loop, and any engine worth considering has it.

Paged KV cache

The naive allocation reserves a contiguous block per sequence sized for the maximum possible length. A sequence that could reach 128k tokens but stops at 2k has reserved — and wasted — the remaining 98%. Multiplied across a batch, the wasted memory is the reason the batch could not be larger.

Paged allocation stores the cache in fixed-size blocks with an index per sequence, exactly as virtual memory does. Blocks are allocated as the sequence grows and freed when it ends, and internal fragmentation is bounded by the block size rather than by the maximum length. The approach was introduced as PagedAttention in the vLLM paper by Kwon and colleagues in 2023 and has since become a standard expectation.

Why it converts into throughput: memory freed from over-reservation becomes KV cache for more concurrent sequences, and more concurrent sequences is more batch. Since it is memory capacity rather than the ridge point that usually limits batch in practice, this feature attacks the actual constraint.

Prefix caching and chunked prefill

• Prefix caching keeps the computed keys and values for a shared prefix so that repeated system prompts, few-shot examples and long documents are not re-prefilled per request. The saving is proportional to the shared fraction, so a workload with a 2,000-token system prompt and 50-token questions saves nearly all of its prefill. Block-based cache layouts make this natural, which is why the two features tend to arrive together.
• Chunked prefill splits a long prompt into pieces and interleaves them with decode steps for other sequences. It slightly slows the prefill and substantially steadies inter-token latency for everyone else — a p99 feature rather than a throughput one, and one whose absence shows up as unexplained latency spikes correlated with other users’ long prompts.

Kernels, graphs and quantisation support

A rough shape holds across generations even as the products change: engines that compile ahead of time for a fixed configuration tend to lead on raw latency and cost more in build time and flexibility, while engines built around a flexible Python runtime tend to lead on scheduling features and breadth of model support. Which of those you want is a property of your deployment, not of the engines.

The benchmarking protocol

If you are going to compare engines, this is the minimum for the result to mean anything. Publish the method alongside the numbers — partly so others can check it, and mostly because writing the method down is what exposes the confounder you were about to miss.

• Hold everything constant but the engine. Same hardware, same driver, same model checkpoint, same weight precision, same KV precision, same maximum context, same tensor-parallel degree. A precision difference alone can produce a 2× gap.
• Replay your own length distribution. Sample input and output lengths from production logs. A uniform 512-in / 128-out synthetic benchmark measures a workload nobody has.
• Sweep concurrency, do not pick one. The answer is a curve of throughput against p95 inter-token latency as concurrency rises. A single point hides the knee, and the knee is the decision.
• Warm up, then measure a steady state. Compilation, graph capture, cache warming and autotuning all happen once. Include them if you restart often; exclude them if you do not, and say which.
• Report percentiles and the token count. Means hide the tail that users experience, and throughput without a token accounting is not reproducible.
• Disable or equalise features that change the answer. Prefix caching on a repetitive benchmark can multiply throughput. If one engine has it enabled and the other does not, you measured the feature and not the engine.