Latency Spikes That Aren’t the Model

“The model got slow” is a conclusion, not an observation. A request has at least six phases and the model is responsible for two of them. Splitting the number is fifteen minutes of work and it routinely relocates the problem to connection setup, a queue you wrote, or an event loop that is blocked.

One number is six numbers

Two structural facts fall out of that table. Time to first token and tokens per second behave differently and have unrelated fixes, so averaging them into “latency” guarantees you optimise the wrong one. And the first three phases together can exceed the model’s contribution entirely on a short request, which is why connection reuse is the highest-value latency fix most teams have not yet made.

Measuring the phases

curl reports the phase boundaries directly, with no instrumentation required. Run it twice: once cold, once against a warm connection, and the difference is what pooling would save you.

curl -sS -o /dev/null \
  -w 'dns      %{time_namelookup}s\ntcp      %{time_connect}s\ntls      %{time_appconnect}s\nsent     %{time_pretransfer}s\nfirstbyte %{time_starttransfer}s\ntotal    %{time_total}s\n' \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d @request.json \
  https://api.example-provider.com/v1/chat/completions

The values are cumulative from the start of the request, so the phase durations are the differences between adjacent lines. A time_appconnect of 0.4s on every single request is the signature of a client being constructed per call.

In application code, measure the same boundaries plus the two that curl cannot see — the time your own code spent before the request left, and the gap between chunks.

import time

t0 = time.perf_counter()
stream = client.chat.completions.create(**kwargs, stream=True)
t_open = time.perf_counter()          # request accepted, headers back

first = None
last = t_open
gaps = []
n = 0
for chunk in stream:
    now = time.perf_counter()
    if first is None:
        first = now                   # true time to first token
    else:
        gaps.append(now - last)
    last = now
    n += 1

print(f"setup+ttfb {first - t0:.3f}s  stream {last - first:.3f}s "
      f"chunks {n}  max_gap {max(gaps or [0]):.3f}s")

max_gap is the field worth keeping. A healthy stream has gaps in the tens of milliseconds; a single gap of several seconds mid-stream is a stall, and that is a different page rather than a latency problem.

The causes that are on your side

1. No connection reuse. Constructing a new HTTP client per request pays DNS, TCP and TLS every time — commonly 100–300ms of pure overhead on a cross-region call, on a request whose useful work might be 400ms. Build the client once at module scope and reuse it. In serverless, hoist it outside the handler so it survives between invocations. This is the single most common finding on this page.
2. A queue you wrote. A semaphore, a thread pool, a worker concurrency limit, a database connection pool. Under load, time is spent waiting for a slot and attributed to the model. Test: record the timestamp when the request is enqueued as well as when it is sent. If the gap is non-zero, you have found it, and no provider change will help.
3. A blocked event loop. In async code, one synchronous call — a blocking HTTP client, a big JSON parse, a tokeniser running on the main thread — stalls every concurrent request. The signature is that latency degrades with concurrency while the provider’s own timings stay flat.
4. Retries you cannot see. Most SDKs retry automatically. A silent retry doubles or triples the observed latency and looks like one slow request. Turn client-side retries down to zero temporarily, or log attempts, before believing any latency figure.
5. DNS in containers. Misconfigured resolvers, aggressive search-domain lists that try several suffixes before the real one, and no caching layer all add up. A time_namelookup above a few milliseconds on a repeated call is diagnostic.
6. Prompt growth. Prefill scales with input length, so a retrieval change that doubles the prompt raises time to first token even though nothing about the model changed. Correlate latency against prompt_tokens before blaming the provider.

The causes that are on theirs

• Queueing at peak. Shared capacity means your time to first token depends on other people’s traffic. The signature is a daily or weekly shape in the p95 that matches business hours in a particular region.
• Cold starts. Less-used models, and dedicated or self-hosted deployments that scale to zero, pay a startup cost on the first request. The signature is a bimodal distribution: most requests fast, a minority very slow, with nothing in between. Cold starts covers the mechanism.
• A model or route change. An alias that moved to a larger model is slower per token by construction. Group your latency by the resolved model field before concluding that a single model got slower.
• Reasoning tokens. If a reasoning effort setting moved, time to first visible token rises sharply while throughput is unchanged. The usage object will show it; the latency number alone will not.
• Geography. A round trip between continents is tens of milliseconds at the speed of light and rather more in practice, paid once per round trip — which is another argument for connection reuse, since a cold TLS handshake pays it several times.

Averages hide the spike you are chasing

A mean latency is close to useless for this work. If 1% of requests take 30 seconds and the rest take 500ms, the mean is about 800ms — a number that describes no request anyone experienced, and one that will not move enough to alert on when the tail doubles.

SELECT date_trunc('hour', ts) AS hour, model,
       percentile_cont(0.50) WITHIN GROUP (ORDER BY ttft_ms) AS ttft_p50,
       percentile_cont(0.95) WITHIN GROUP (ORDER BY ttft_ms) AS ttft_p95,
       percentile_cont(0.99) WITHIN GROUP (ORDER BY ttft_ms) AS ttft_p99,
       avg(completion_tokens / nullif(stream_ms, 0) * 1000)  AS tok_per_s
FROM llm_requests
WHERE ts > now() - interval '3 days'
GROUP BY 1, 2 ORDER BY 1;

Read p50 and p95 as separate signals. A p50 that moved is a systematic change — a bigger prompt, a different model, a slower route. A p50 that is flat with a p95 that doubled is contention or a bad minority: cold starts, one unhealthy instance, or a subset of requests that are much larger than the rest. Latency percentiles goes further on what to alert on.