Streaming Partial Thoughts: UX for a Model That Thinks for 40 Seconds

A forty-second wait with no output is indistinguishable from a hang. The engineering fix and the interface fix are the same fix — put bytes on the wire — but what you do with those bytes once they arrive is a design decision with several bad answers.

What the user is actually experiencing

Three separate things go wrong in a long silence, and they need different answers.

• Uncertainty about liveness. Is it working or is it broken? A spinner answers this and nothing else. It is also a lie after about ten seconds, because a spinner implies imminence.
• Uncertainty about duration. Should I wait or come back? Nothing in a spinner addresses this. An elapsed-time counter does, honestly and cheaply.
• Loss of context. By the time the answer arrives the user has switched tabs and forgotten the question. This is an argument for restating the question near the answer, and for a notification if the wait can exceed a minute.

The events on the wire

Where a provider streams the trace, it arrives as its own event type rather than mixed into the answer text. Anthropic’s streaming format emits a content block of type thinking whose deltas are thinking_delta rather than text_delta; the answer follows as a separate block. OpenAI’s Responses API emits reasoning summary events distinct from the output text events. The shapes differ, the discipline does not:

event: content_block_start
data: {"index":0,"content_block":{"type":"thinking","thinking":""}}

event: content_block_delta
data: {"index":0,"delta":{"type":"thinking_delta","thinking":"Let me check "}}

event: content_block_delta
data: {"index":0,"delta":{"type":"thinking_delta","thinking":"the second constraint"}}

event: content_block_stop
data: {"index":0}

event: content_block_start
data: {"index":1,"content_block":{"type":"text","text":""}}
...

for await (const ev of stream) {
  switch (ev.type) {
    case "content_block_delta":
      if (ev.delta.type === "thinking_delta") ui.thinking(ev.delta.thinking);
      if (ev.delta.type === "text_delta")     ui.answer(ev.delta.text);
      break;
    case "content_block_start":
      if (ev.content_block.type === "text") ui.thinkingDone();
      break;
  }
}

The one rule that matters: route the two into different sinks. Code that concatenates every delta into one buffer will render the model’s internal monologue as though it were the reply, and users will read it as the reply — including the false starts and the abandoned wrong answers, which is a worse outcome than the silence you were trying to fix.

Four patterns that work

Whichever you pick, style the trace so it cannot be mistaken for the answer — lower contrast, smaller, indented, visually subordinate. And keep it out of the accessibility live region: a screen reader announcing a thousand tokens of internal monologue before the answer is unusable. Announce the state change, not the content.

Update rate and layout

Deltas arrive far faster than anyone can read them, and rendering every one is both wasteful and unpleasant to look at. Batch them on an animation frame, or on a fixed interval of 50 to 100 ms, and append rather than re-rendering the container. If you are showing a rolling line, give it a fixed height: text that grows and shrinks pushes the rest of the page around, and a user reading something else on screen will lose their place several times a second.

The other reason to batch is that thinking output is not smooth. Long runs arrive in bursts as buffers flush, so a character-by-character animation applied on top either lags seconds behind the real state or catches up in a jarring rush. Render what has arrived and let the stream do its own pacing.

Finally, decide what happens to the trace after the answer lands. The usual right answer is to collapse it automatically and keep it available, because a completed trace is clutter for most readers and the single most useful artefact for the one who wants to know why. In a multi-turn transcript, collapse all previous turns’ traces by default — a conversation where every turn carries a thousand tokens of visible monologue becomes unreadable by the third exchange.

Cancellation, and what you still pay

Give the user a cancel button, always. A request that will take ninety seconds needs an exit, and abandoning the tab is not one — it leaves your server generating tokens into a void.

Two things to be clear about. Aborting the HTTP request stops the stream at your end; you are still billed for the tokens that were generated before the abort took effect, because they were generated. Cancellation is a latency and attention feature, not a refund. And on most stacks an abort must be propagated all the way to the provider connection — a client-side AbortController that only unhooks your own listener leaves the upstream request running to completion, which is the expensive version of the bug.

Log cancellations with the elapsed time at cancel. That distribution tells you what your real latency budget is, which is usually shorter than the one anyone specified, and it feeds directly into the timeout decisions in latency budgets.

When there is nothing to stream

If the provider hides the trace entirely, you have no partial content to show and the collapsed-trace pattern has nothing in it. Three options remain, in order of preference: show elapsed time and a cancel button and say plainly that the model is working; stream something you generated yourself, such as the retrieval step or the plan, so the wait is populated with real progress from another part of the pipeline; or move the whole thing off the request path and notify on completion.

What not to do is fake it. Rotating status messages that are not tied to anything the model is doing — “consulting sources”, “considering alternatives” — are invented, and users work this out quickly. An honest counter is better received than a fictional narrative, and it does not need maintaining when the pipeline changes.