What Happens Between Your Request and the First Token

“The API is slow” is never a diagnosis. Between your process calling fetch and the first byte of content arriving there are seven or eight distinct hops, they fail and stall for unrelated reasons, and only about half of them are the model’s. This page names them all so you can say which one is slow.

The path, hop by hop

Assume a normal chat completion over HTTPS to a hosted provider. In order:

• Serialisation and DNS/TCP/TLS. Building the JSON body, resolving the host, the TCP handshake and the TLS handshake. On a warm connection pool this is nearly free; on a cold one, TLS alone is a round trip or two, which on a transatlantic path is not noise.
• Network to the edge. One one-way trip of your request bytes. Bounded below by the speed of light in fibre, so no amount of engineering removes it — only moving closer does.
• Gateway work. Auth, key lookup, spend checks, request validation, moderation pre-filters, routing to a backend. Milliseconds when it is a cache hit, considerably more when any of it touches a database.
• Admission and queueing. The request joins a scheduler’s waiting list until the server has room in its running batch. This is the hop with the highest variance by a wide margin, and it is invisible from outside.
• Prefill. The prompt goes through the model in one compute-bound pass, populating the KV cache. Grows with prompt length.
• First decode step. One more forward pass produces the first output token.
• Detokenisation, framing and the trip back. The token becomes text, gets wrapped in an SSE frame, and travels back to you. Any proxy that buffers rather than streams silently adds the whole rest of the generation to this hop.

Everything after that is the decode loop repeating, one token per iteration, until a stop condition.

The only equation on this page

Time to first token is the sum of those hops, and total response time adds the rest of the decode:

ttft   = connect + rtt_up + gateway + queue + prefill + decode_1 + rtt_down
total  = ttft + (output_tokens - 1) / decode_rate

It looks trivial. Its value is that it is an identity: every millisecond you experience lives in exactly one of those terms, so “where did the time go” always has an answer. It also makes two things obvious that averaged dashboards hide. A long system prompt moves prefill and nothing else. A verbose answer moves output_tokens and nothing else. Fixing the wrong term produces no improvement at all, which is the usual reason an optimisation “did nothing”.

The hop nobody accounts for

Of the seven hops, six are roughly stable for a given request shape. The queue is not. A modern inference server runs many sequences concurrently and admits new ones between decode iterations; when it is saturated, your request waits, and the wait depends entirely on other people’s traffic. Nothing about your request changed. This is why the same prompt to the same model can be an order of magnitude apart on two consecutive calls, and it is the single largest reason that a mean latency figure is useless here — see continuous batching for the scheduler that causes it.

The practical consequence: you cannot fix queueing from the client. You can only avoid it, by having somewhere else to send the request.

Which hops are yours

Filling in your own numbers

The honest version of a time budget is one you produce from your own traffic, because every term above depends on your region, your prompt shape and your provider’s current load. The minimum instrumentation that makes the identity usable is three timestamps per request: when you called, when the response headers arrived, and when the first content chunk arrived. Headers-minus-call is roughly connect plus network plus gateway; first-chunk-minus-headers is queue plus prefill. Record output token count and total duration and you can back out the decode rate.

Three timestamps and a token count is enough to tell you which of the seven hops owns your p95, and that is the whole point of the exercise. Record them as a distribution rather than a mean — a single cold start drags an average past every request anybody actually experienced.

One caution when you compare your numbers against anything the provider reports. A server-side duration measures from when the request reached the server; your client-side duration includes the network in both directions and any time your own process spent between deciding to make the call and actually making it. On a saturated Node process with a busy event loop, that last gap is real and is invisible to both timers. If the two figures disagree by a consistent margin, the margin is the answer rather than the discrepancy.

What a slow p95 usually turns out to be

Once the identity is instrumented, the same handful of causes account for most investigations, and they are worth checking in roughly this order because the cost of checking rises as you go down:

• A buffering hop you own. If time to first token is suspiciously close to total duration, nothing is streaming and the model is not the problem. Check the proxy, the CDN and any framework helper that collects a response body before sending it.
• A new connection per request. A client constructed inside the handler rather than at module scope gets no connection reuse, and pays TLS on every call. This is a one-line fix that regularly removes a hundred milliseconds or more.
• A prompt that grew. Prefill scales with prompt length, and prompts grow by accretion — one more example, one more retrieved chunk, one more instruction. Plot p95 time to first token against prompt tokens and the relationship is usually visible immediately.
• A cache that stopped hitting. A prefix cache is silently invalidated by a single volatile token near the front of the prompt, and the only symptom is the prefill term getting worse. Log the reported cached-token count so this shows up as a metric rather than as a mystery.
• The queue, which is not yours. If none of the above holds and the term that moved is the gap between headers and first content, you are waiting for admission, and the only remaining lever is where you send the request.