Skip to content
Multigrid

Inference, serving & latency

What actually happens when a request reaches a model server, and how to reason about latency, throughput, limits and failures instead of guessing at them.

Inference is the only part of a language model you interact with, and it is the part most explanations skip. They stop at “the model generates a token at a time” and leave you without an answer to the questions you actually have: why the same request is sometimes ten times slower, what a sensible timeout is, why the provider charges four times as much for output as for input, and how much memory one open conversation is costing somebody.

These pages answer those from the serving side. Most of them are arithmetic — the numbers here are derivable from architecture and hardware specifications, and where they are, the working is shown rather than the conclusion asserted. Where something depends on a scheduler you cannot see, the page says so instead of inventing a figure.

What Happens Between Your Request and the First Token

The full path from an HTTP request to the first streamed byte, and which of the hops along it you can actually change.

5 min read

Prefill vs Decode: The Two Halves of Inference

Why reading a prompt and writing an answer are different computations on different hardware bottlenecks, derived from arithmetic intensity.

5 min read

KV Cache: The Memory That Makes Generation Affordable

What the KV cache stores, the exact formula for its size per token, and worked examples of how many concurrent users fit in a GPU.

5 min read

Continuous Batching and Why Your Latency Varies

How iteration-level scheduling shares a GPU between strangers, and why that makes an identical request sometimes an order of magnitude slower.

5 min read

Time to First Token vs Tokens per Second

The two latency numbers behave differently, are fixed by different things, and only one of them matters for any given interface.

5 min read

Speculative Decoding: Two Models, One Output

How a small draft model plus a verification step produces the same distribution as the large model alone, faster — and what governs the speedup.

6 min read

Quantisation for Inference: FP16, INT8, FP8 and INT4

What each numeric format costs in memory and bandwidth, what the published literature claims about quality, and how to choose without trusting either blindly.

5 min read

Is the Quantised Model You’re Calling the Same Model?

A detection procedure you run yourself against any endpoint, to tell whether the model behind a model name changed.

6 min read

Throughput vs Latency: You Can’t Optimise Both

Why batch size trades one against the other, and why the trade gets brutal exactly when the server is busy.

5 min read

Streaming Responses: SSE, Chunks and Backpressure

A correct server-sent-events reader for a chat completion, including the frame handling, the abort path and the billing consequence of hanging up.

6 min read

Cold Starts on Serverless Inference

Where the seconds go when a model has to be loaded before it can answer, estimated from bandwidth, and the four ways the wait gets hidden.

5 min read

Timeouts: Choosing One That Isn’t a Guess

How to derive four separate timeouts from your own latency distribution, and why a single overall deadline is the wrong instrument for a stream.

5 min read

Retries and Backoff for LLM APIs

Full jitter, Retry-After, retry budgets and the hazard that makes retrying an LLM call different from retrying anything else: you may pay twice.

6 min read

Rate Limits: TPM, RPM and How to Live Inside Them

How token buckets actually behave, why token-per-minute limits are charged against an estimate, and a client-side limiter that respects both dimensions.

6 min read

Handling 429s Without Melting Down

Queue, shed or degrade — the three honest responses to being over a limit, with a bounded deadline-aware queue and the arithmetic for sizing it.

6 min read

Batch APIs: Half Price if You Can Wait

What an asynchronous batch endpoint is really selling, which workloads fit inside its window, and the operational differences that decide whether it is worth it.

5 min read

Prompt Caching: How the Major Implementations Differ

Explicit versus automatic caching, the prefix and invalidation rules that decide your hit rate, and the documented numbers as they stood on 3 August 2026.

5 min read

Semantic Caching: Answering Before You Call

Serving an old answer to a similar new question — the calibration harness that sets the threshold, and the cost model that says whether to do it at all.

6 min read

Load Balancing Across Multiple Inference Endpoints

Why round-robin fails for model calls, what latency-aware and cost-aware routing actually optimise, and the cache-affinity tension nobody mentions.

6 min read

Measuring p50, p95 and p99 for LLM Calls

Why the mean is meaningless for a queueing workload, how many samples a p99 actually needs, and the four aggregation mistakes that make a dashboard lie.

6 min read

Other topics

Inference, serving & latency · Multigrid