The Streaming Response Shape a Self-Hosted Llama Server Returns

The model produces one token per forward pass and that is the same everywhere. What differs is the envelope each server wraps it in, and the four common ones disagree on framing, on field names and on how they say they are finished.

The transport underneath all of them

Three of the four use server-sent events: an HTTP response with Content-Type: text/event-stream, held open, carrying lines that begin data: and are separated by blank lines. It is a one-directional stream over ordinary HTTP, which is why it works through proxies that would refuse a WebSocket.

The consequence people trip on is that SSE has no error frame. Once the response has begun with a 200, a failure mid-generation cannot become a 400 — it arrives as a truncated stream, an event in a server-specific error shape, or a connection that simply closes. Client code that only handles errors before the first byte will treat a mid-stream failure as a short answer.

It is worth being clear that streaming changes nothing about how fast the tokens are produced. Generation is sequential either way; streaming only means the server forwards each token as it appears rather than buffering the whole answer. What it buys is that the reader waits for the time to first token instead of for the whole completion, which on a long answer is the difference between a second and half a minute. What it costs is that every consumer of the response now has to deal with partial state — including your logging, your moderation and your error handling.

The OpenAI-compatible shape

vLLM, llama.cpp’s llama-server, TGI and Ollama all expose an OpenAI-compatible /v1/chat/completions, and with "stream": true all four emit chunks of this shape:

data: {"id":"chatcmpl-3f0c","object":"chat.completion.chunk","created":1786000000,
"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,
"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-3f0c","object":"chat.completion.chunk","created":1786000000,
"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,
"delta":{"content":"Reykjav"},"finish_reason":null}]}

data: {"id":"chatcmpl-3f0c","object":"chat.completion.chunk","created":1786000000,
"model":"meta-llama/Llama-3.1-8B-Instruct","choices":[{"index":0,
"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Four properties are worth naming because clients get each of them wrong:

• The first chunk carries delta.role and usually an empty content. Concatenating delta.content blindly requires it to be present; it is not always.
• The terminator data: [DONE] is not JSON. Parsing every data: line as JSON throws on the last one.
• The chunk carrying finish_reason usually has an empty delta, so it is a separate event from the last piece of text.
• Token usage is absent by default. On vLLM you ask for it with "stream_options": {"include_usage": true}, which appends a final chunk with a usage object and no choices.

Chunk boundaries are not token boundaries in any guaranteed way. A server may batch several tokens into one event under load, and a multi-byte character can be split across the tokens that compose it, so decode by accumulating rather than assuming each chunk is independently meaningful text.

Tool calls make the same shape harder in one specific way. When a model calls a function, the arguments stream as a series of delta.tool_calls fragments whose arguments field is a partial JSON string — not partial JSON you can parse, but a string that only becomes valid JSON once every fragment has arrived. Accumulate by index and parse once at the end. Attempting to parse each fragment is the most common cause of a tool-calling client that works non-streaming and breaks the moment streaming is turned on.

Ollama’s native shape is not SSE

Ollama’s own /api/chat and /api/generate stream newline-delimited JSON — one object per line, no data: prefix, no blank-line separators and no [DONE] sentinel:

{"model":"llama3.1","created_at":"2026-08-11T09:00:00Z",
 "message":{"role":"assistant","content":"Reykjav"},"done":false}
{"model":"llama3.1","created_at":"2026-08-11T09:00:00Z",
 "message":{"role":"assistant","content":"ik"},"done":false}
{"model":"llama3.1","created_at":"2026-08-11T09:00:01Z",
 "message":{"role":"assistant","content":""},"done":true,
 "done_reason":"stop","total_duration":1290000000,
 "prompt_eval_count":31,"eval_count":58}

The stream terminates on done: true, and that final object is where the counts live: prompt_eval_count and eval_count are the input and output token counts, and the duration fields are nanoseconds. An SSE client pointed at this endpoint sees no events at all, because there are no data: lines to parse — a failure that looks like a hang.

llama.cpp and TGI native endpoints

llama.cpp’s native /completion is SSE, but with its own field names — the text is content at the top level and completion is signalled by a boolean:

data: {"content":"Reykjav","stop":false}

data: {"content":"","stop":true,"stopped_eos":true,"stopped_word":false,
"stopped_limit":false,"tokens_predicted":58,"tokens_evaluated":31}

The three stopped_* booleans are more informative than a single finish reason: they distinguish an end-of-sequence token from a matched stop string from hitting the prediction limit, which is precisely the distinction you need when diagnosing a stop sequence that seems to be ignored.

Text Generation Inference’s native /generate_stream is SSE again, with a per-token object and the full text only at the end:

data: {"index":1,"token":{"id":49444,"text":"Reykjav","logprob":-0.31,
"special":false},"generated_text":null,"details":null}

data: {"index":2,"token":{"id":1609,"text":"ik","logprob":-0.02,
"special":false},"generated_text":"Reykjavik","details":
{"finish_reason":"eos_token","generated_tokens":58,"seed":null}}

The special flag is the useful field here: it marks tokens such as <|eot_id|>, so a client can drop them rather than printing them into the user’s output. Where a stack does not offer that flag, control tokens leaking into rendered text is a recognisable symptom of a template or stop-token mismatch rather than of streaming itself.

Writing a client that survives all four

Most of the bugs in streaming clients are not about the model at all. They are about parsing, and they follow a short list:

1. Buffer by delimiter, not by chunk. A TCP read is not an event. Accumulate bytes and split on the delimiter for the format — a blank line for SSE, a newline for Ollama’s ndjson — and keep the remainder for the next read. A client that treats every read as one complete event works perfectly on localhost and fails behind a proxy.
2. Handle the sentinel before parsing. Check for [DONE] as a string, then parse. In reverse order, every stream ends in an exception.
3. Treat missing fields as absent, not as errors. delta.content is missing on the role chunk and on the finish chunk, and may be missing on a chunk that carries only a tool-call fragment.
4. Decode text incrementally with a stateful decoder. A UTF-8 sequence can be split across chunks. Decoding each chunk independently produces replacement characters in the middle of non-Latin text — a bug that never appears in English testing.
5. Set a read timeout, not just a request timeout. A stalled generation holds an open connection indefinitely. The signal you want is “no event for N seconds”, and note that some servers send keepalive comment lines (a line beginning : in SSE) that your parser must ignore without treating them as data.
6. Decide what a mid-stream failure means to the user. You have partial text and no ending. Retrying regenerates from scratch and costs the prefill again; showing the partial answer is usually honest if you mark it as incomplete. The one thing not to do is present it as finished.

What the last event tells you

Whichever shape you are consuming, read the terminal event rather than discarding it. It answers the question that the text alone cannot: did the model finish, or did it run out of room? An OpenAI-compatible finish_reason of "length", llama.cpp’s stopped_limit, TGI’s finish_reason of "length" and Ollama’s done_reason of "length" all mean the same thing — the output cap cut it off — and an application that ignores them will show a truncated answer as if it were complete. That is the difference between a bug report and a retry with a larger output limit.