Setting Expectations: Telling Users What AI Can’t Do

Users do not complain that a model is imperfect. They complain when it fails in a way they had no way of predicting — and almost every one of those is a specific, nameable mechanical limit that nobody told them about.

Expectation failure is a mental-model bug

Norman’s framing in The Design of Everyday Things is the right one here: a user operates a system through a conceptual model assembled from whatever the interface makes visible. When the system’s actual behaviour and that model diverge, the user is not confused about the world, they are correctly applying a model that the interface gave them.

A chat box is unusually bad at supplying a model. It looks like a person, it is presented in the visual language of a messaging app, and it has no visible mechanism at all — no fields, no settings, no indication of what it can reach. Everything a user believes about it, they inferred. So expectation-setting is not a politeness exercise; it is the only channel through which any conceptual model gets in.

Three classes, three mechanical causes

Nearly every “the AI is broken” report falls into one of three buckets, and each one traces to a property you can state exactly.

Capability: it thinks it can do a thing it cannot

The user asks about last week’s release, a live share price, or the contents of a URL, and gets a fluent answer built out of nothing. The cause is the knowledge cutoff plus the absence of tools: without retrieval or browsing, the model has no route to anything after training, and it has no mechanism for noticing that it has no route. The same class covers arithmetic and counting — a model doing arithmetic is generating plausible digits, not computing.

Scope: it thinks the model read everything

A user uploads a 300-page contract and asks a question about clause 214. If the document was truncated, chunked and partially retrieved, the model answered from a fragment. Nothing in the answer says so. The causes are the context window and what gets truncated, and the related failure where material is present but ignored — attention to the middle of a long context.

Permanence: it thinks the model remembers

“I told it my name yesterday.” The model is stateless; a conversation persists only because your product resends it. Anything outside what you resend does not exist. This one is worsened by products that have some memory, because a user who has seen it remember once will reasonably assume it always does.

Writing the line

Three rules, and they are all consequences of the class above rather than style preferences.

• Name the boundary, not the vibe. “May produce inaccurate information” is unfalsifiable and teaches nothing. “Doesn’t know about anything after March 2025” and “Only sees the files you attach to this message” are statements a user can act on, because each one implies a different next move.
• Say what to do instead. A limit with no alternative is a dead end. “Can’t open links — paste the text and I’ll work from that” converts a failure into a workflow.
• Do not apologise in the model’s voice. If the limit is your product’s (you did not wire up retrieval), say it as your product. Letting the model improvise an explanation of its own limits produces confident, wrong descriptions of your architecture.

Where the line goes

The single most common mistake is putting all of this in onboarding. A limit explained on day one is being recalled on day forty, at the moment of a task, by someone who is not thinking about your onboarding modal. Expectation-setting has to be co-located with the failure it predicts:

The hedging tax

There is a real cost to over-warning, and it is well documented outside this field. Wogalter’s Communication–Human Information Processing model — the standard framework in warning science — treats a warning as a chain: it has to be noticed, encoded, understood, believed and then acted on, and the chain breaks at any link. Habituation attacks the first link directly: a warning that appears on every instance stops being attended to, and once attention is lost none of the later stages happen at all.

The same logic governs the model’s own hedging. An assistant that prefixes every answer with a paragraph of qualification is training users to skip the first paragraph, which is exactly where you would want a genuine caveat to go. If you can only afford to be believed a few times, spend it on the cases where being wrong is expensive.

The practical consequence is a budget rather than a rule: decide how many caveats a user will see per session, and spend them on the limits that are both non-obvious and consequential. A limit that the interface can simply enforce — greying out an action the model cannot do, refusing an upload that will be truncated — costs nothing from that budget, which makes enforcement strictly better than explanation wherever it is available.

There is a second reason to prefer enforcement, and it is about who bears the cost of being wrong. A warning transfers responsibility to the user: we told you, you proceeded. That is a reasonable arrangement only when the user had the information to act on it, and in almost every case discussed on this page they do not — they cannot see how much of the document was read, whether retrieval returned anything, or which of their two questions the model answered. A caveat used as a liability transfer, rather than as a genuine input to a decision the user is equipped to make, is a warning that was never intended to change anything.

The test worth applying to each caveat before it ships is therefore narrow: name the decision it changes. “Only sees the files you attach” changes whether the user attaches a second file. “May produce inaccurate information” changes nothing, because it offers no alternative course of action and applies equally to every output. If a line cannot pass that test, it is not expectation-setting; it is a disclaimer, and it should be somewhere a disclaimer belongs rather than spending attention in the interface.