Loading States for Slow AI: The 40-Second Problem

The interaction research that underpins loading design was written for a world where three seconds was a catastrophe. A reasoning model with a retrieval step in front of it can take forty. Every rule of thumb you inherited breaks at that scale, and the replacements have to be derived rather than looked up.

The three response-time limits

Jakob Nielsen’s Usability Engineering (1993) sets out three thresholds, themselves drawing on Robert B. Miller’s 1968 work on response time in conversational transactions. They have held up for three decades because they are grounded in human perception rather than in the hardware of any era:

• 0.1 second — the limit for the result to feel instantaneous, as if the user caused it directly.
• 1 second — the limit for uninterrupted flow of thought. The user notices the delay but does not lose their place.
• 10 seconds — the limit for keeping attention on the task. Beyond it, users start doing something else, and the interface has to be designed for their return rather than their attention.

Now put real numbers against them. Time to first token for a large model is typically a second or more before you have added anything; prefill grows with prompt length, so a long system prompt and a retrieved context push it further; a reasoning model spends hidden thinking tokens before the first visible one; and a p95 several times the p50 is ordinary, not pathological — which is the whole reason latency is reported as percentiles. An agent loop with three tool calls in it is comfortably past the third threshold on a good day.

So the design question is not “how do we stay under a limit”. It is “we are past the attention limit; what does the interface owe the user for the next thirty seconds”.

Why a spinner is not a design

An indeterminate spinner carries exactly one bit: something is happening. That is adequate for 800 milliseconds and useless at thirty seconds, for a reason that is structural rather than aesthetic — a spinner that has been going for forty seconds is indistinguishable from a spinner attached to a dead request. The user cannot tell a slow success from a hung socket, so the rational move is to reload, which starts a second billed generation and abandons the first.

The obvious fix — a determinate progress bar — is not available. You do not know how many tokens the answer will be. Nothing knows: the model decides when to stop as it goes. A percentage would be an invention, and there is no honest way to derive one from tokens emitted so far. Which leaves the interesting question: what can be shown that is both informative and true?

Progress you can show without lying

Four things are genuinely knowable during a wait, and all four are worth more than a percentage:

Note the pattern: each signal is a fact the system already has. The moment a progress display requires an estimate the system cannot make, it has crossed from communication into decoration, and users learn to discount it.

The wait-state machine

The states escalate with elapsed time, because what the user needs at two seconds and at twenty-five seconds are different things. Written as a machine so it can be implemented once and reused:

idle
  -> submit -> pending(t=0)

pending:
  t < 0.4s   : nothing. Rendering a spinner for a fast response
               produces a flash that reads as a glitch.
  0.4 - 3s   : quiet indicator in place. Reserve the final layout
               box now so nothing shifts when text arrives.
  3 - 12s    : + elapsed timer, + phase label if phases are real.
               Cancel becomes visible.
  12 - 30s   : + cancel promoted to a primary control.
               Copy acknowledges the wait honestly.
  > 30s      : + offer the asynchronous path: "keep working, we'll
               tell you when it lands". Do not silently keep waiting.

  first token -> streaming
  error       -> failed        (see the error taxonomy)
  user cancel -> cancelled     (partial output retained)

streaming:
  stalled > 10s with no token -> degraded (stream is alive but empty)
  finish  -> done

The one at 30 seconds is the state most products skip and the one that matters most. Past roughly half a minute the honest design is not a better spinner, it is a different interaction: hand the work to a background job, free the interface, and notify on completion. See when to move generation off the request path.

Cancel is a latency feature

Cancel is usually filed under error handling. It belongs here, for three reasons that all come from the model’s mechanics.

• It restores agency during a wait. The user who cannot stop a forty-second generation is trapped; the one who can stop it is waiting by choice. The behaviour of the system is identical and the experience is not.
• It is a spending control. Aborting a stream stops generation, and generation is the expensive half — how an aborted stream is billed varies by provider, but nobody bills you for tokens after you stopped asking for them.
• It prevents the double-charge reload. Without a cancel, the escape hatch users find is refreshing the page, which abandons the first generation without stopping it and starts a second.

Cancel must retain whatever streamed before it. A cancel that wipes the partial answer punishes the user for using it, and they will not use it twice.

One implementation note that decides whether any of this is real: aborting the client request has to actually abort the upstream one. Closing an EventSource or dropping a fetch reader stops your process reading, and unless the abort signal is propagated all the way to the provider call the generation continues, is billed in full, and the only thing you cancelled was your own attention. This is the sort of bug that never surfaces in testing and appears on an invoice.

Finally, resist the urge to make the wait more entertaining. Rotating messages, fake progress and animated mascots are all attempts to buy patience with novelty, and they wear out on the second exposure while the underlying wait does not. Every item in the state machine above is information the user did not have; nothing in it is decoration, and that is the property to preserve when the design gets revisited.