Streaming vs Non-Streaming: Any Cost Difference?

For the same prompt producing the same answer, streaming and non-streaming cost the same. The question is worth a page anyway, because the exception is real, is worth money, and is not documented consistently.

The short answer

Set stream: true and nothing about your bill changes. You are billed on tokens processed and tokens generated. Streaming is a change to how those generated tokens reach you — one server-sent event per chunk rather than one JSON body at the end — and the model does exactly the same work either way.

If someone claims streaming is cheaper or more expensive, ask which meter they think it changes. There isn’t one. What streaming changes is perceived latency: the reader starts at time to first token instead of at completion. That is a large user-experience win and belongs in a different document from the cost proposal.

Why transport cannot change the price

Generation is autoregressive — one forward pass per token, sequential, regardless of what happens to the token afterwards. In the non-streaming case the server buffers each token and sends the lot at the end. In the streaming case it forwards each one as it appears. The arithmetic on the accelerator is identical, and the price is a function of that arithmetic.

The one wrinkle worth knowing is that in streaming mode the usage object arrives at the end of the stream, in a final chunk, and some clients need to opt into receiving it. If your accounting reads usage from the response body, switching a route to streaming can silently drop your token counts to zero — which looks like a cost reduction on a dashboard and is a metering bug. That is the most common way streaming appears to change costs.

The one case where it does differ

A user closes the tab halfway through a long answer. What have you paid for?

There are two possibilities and they differ by a large factor. If the disconnect propagates and the provider stops generating, you are billed for the tokens produced up to that point. If it does not — if the provider completes the generation into the void — you are billed in full.

abort honoured:
  cost = ( T_in * P_in  +  a * T_out * P_out ) / 1e6
     a = fraction of the answer generated before the abort

abort not honoured:
  cost = ( T_in * P_in  +  T_out * P_out ) / 1e6

Assumed: T_in 2,000, T_out 1,200, P_in $1/M, P_out $5/M,
abandon rate 15%, mean a = 0.35 on abandoned requests.

honoured     : 0.85*(0.002+0.006) + 0.15*(0.002+0.35*0.006)
             = 0.0068 + 0.15*0.00410 = $0.00742
not honoured : 0.0080
difference   : 7% of the endpoint's bill

Seven percent, on assumed numbers, for a behaviour you did not choose and probably have not checked. It is larger on anything with a high abandon rate and long answers — a chat interface where users read the first sentence and rephrase is the archetype.

Non-streaming has no equivalent lever. If you abandon a non-streamed request, the server was going to generate the whole answer regardless; there is no partial state to stop at. So the honest framing is not “streaming is cheaper” but “streaming is the only mode in which stopping early is possible at all”. Whether your stack takes that opportunity is a property of your stack.

The test to run

This is measurable in twenty minutes, on your provider, through your proxy, with your HTTP client, which is the only configuration whose answer applies to you. Every layer between you and the provider — Cloudflare, a load balancer, a gateway, a serverless runtime, an SDK — can swallow a cancellation, so the answer for a bare curl is not necessarily the answer for your application.

1. Pick a prompt that reliably generates ~1,000 tokens.
2. Control arm: 10 streamed requests read to completion.
   Record reported cost / completion_tokens for each.
3. Test arm: 10 streamed requests where the client aborts
   after roughly 100 tokens. Abort the way your app aborts
   (AbortController, cancelled context, closed response).
4. Wait for usage to settle, then compare the two arms in
   the provider's own usage reporting, not in your client --
   an aborted stream may never deliver its usage chunk.

Test arm ~= 10% of control  -> aborts are honoured.
Test arm ~= control         -> you pay in full; the client
                               abort is doing nothing.

Step 4 is the one people get wrong. The usage object normally rides in the final stream chunk, and an aborted stream by definition never receives it, so your own logs will show nothing and prove nothing. The provider’s usage records are the source of truth for this experiment.

If the result is “not honoured”, the mitigations are ordinary: keep max_tokens tight so the worst case is bounded, and shorten answers so there is less to abandon. Both are covered in output length control.

Costs that are not token costs

Streaming is free at the provider and not quite free everywhere else.

• Connection time. A streamed response holds a connection open for the duration of the generation. On per-second serverless billing, thirty seconds of held connection is thirty seconds of compute you pay for, and concurrency limits are consumed for longer.
• Buffering proxies undo it. A proxy that buffers the response defeats the latency benefit while keeping all of the costs — you get held connections and no faster first token. This is a common misconfiguration and the symptom is that streaming “works” but arrives all at once.
• Accounting is harder. Token counts must be accumulated from the stream rather than read from a field, so metering code has one more failure mode. See the note in the previous section about routes that silently stop reporting usage.
• Response caching is harder. Caching a streamed response for reuse means reassembling it, which is easy to skip — and skipping it forfeits a saving much larger than anything else on this page.

None of which is an argument against streaming. It is an argument for knowing which of these your architecture has, so that when the bill is reviewed the answer to “does streaming cost more” is a number rather than a shrug.

There is one genuine cost saving that only streaming makes available, and it is worth building deliberately rather than hoping for. Because tokens arrive as they are produced, your server can inspect them mid-flight and stop. If the model has started a repetition loop, has begun a section your schema does not have a place for, or has already produced the JSON object you asked for and is now adding commentary, you can cancel at that point instead of paying for the rest. That is a server-side decision based on content, it is entirely under your control, and it does not depend on whether client disconnects propagate — which makes it more reliable than the abandon case above and, on a route with a long tail of over-long answers, worth more.

The counterpart is that server-side streaming makes it easy to build a proxy that holds the response open and never cancels anything, and to then be surprised that costs did not move. Streaming is a capability, not an optimisation; it pays only where something acts on the tokens as they arrive.