The Support Load an AI Feature Creates
8 min read · updated August 4, 2026
An AI feature generates a support load that is unlike the rest of the product’s, and the reason is structural: the feature is non-deterministic, the user cannot tell a limitation from a bug, and the support agent usually cannot reproduce what happened. Classifying the load by cause rather than by symptom is what turns it into a list of product changes.
Why this load is different
Three properties, and each has a direct consequence for the queue.
- It is not reproducible on demand. The same input can produce a different output, so “steps to reproduce” is not available. Without the stored request and response, the agent has only the user’s description of an output neither of them still has.
- The user cannot classify the failure. “It gave me the wrong answer” covers a documented limitation, a retrieval fault, a prompt-injection attempt by a third party, and a genuine defect. Ordinary software gives the user error messages that partition the space; this does not.
- Expectations were set by something else. Users arrive with expectations formed by consumer chat assistants, so the baseline they compare against is not your product and not your documentation.
The six categories
| Category | Description |
|---|---|
| 1 · Capability mismatch | The user asked for something the feature was never meant to do. Not a bug, and not the user's fault either — it is a scoping failure in the interface, which presented an open text box and thereby promised everything. |
| 2 · Correct output, wrong expectation | The output is right by the product's definition and wrong by the user's. Common with summarisation and extraction, where 'complete' means different things to the two parties. |
| 3 · Genuinely wrong output | The model was wrong, or the retrieval was. The only category that model or pipeline work fixes, and usually the minority. |
| 4 · Output shape problems | The answer is correct and unusable: truncated, in the wrong format, in the wrong language, too long for the field it lands in. Fully preventable and frequently the largest category at launch. |
| 5 · Trust and disclosure | 'Was I talking to a person?', 'Is my data being used for training?', 'Why did it know that about my account?'. These arrive as support tickets and are answered by the interface, the privacy page and the contract. |
| 6 · Cost and limits | Anything about quotas, rate limits, credits or unexpected spend. Distinct from the rest because the answer is always an account fact, and because these are the tickets that escalate fastest. |
Most of them are interface faults
The useful reading of that table is that only category 3 is fixed by working on the model. Categories 1, 2 and 4 are fixed by narrowing what the feature offers and being explicit about what it produces; 5 is fixed by disclosure; 6 is fixed by making limits visible before they bite.
That is why teams that respond to support load by improving the model see the queue barely move. The model was rarely the constraint. An open text box is a promise that the feature can do anything, and every ticket in category 1 is that promise being tested.
Expectation setting and error UX for AI features cover the interface side; the removal decision, if it comes to that, is in when to remove a chatbot.
Instrumenting before launch
A ticket that cannot be tied to a request is a ticket that produces an apology rather than a fix. Five things must exist before launch, and all five are cheap if built before and expensive if retrofitted.
- A request id surfaced in the interface. Short, copyable, shown next to every AI-generated output. This single change converts an unreproducible ticket into a lookup.
- The assembled request stored, not the template. The full message array as sent, including retrieved context and tool schemas. Diffing a failing request against a working one is the whole diagnosis, and it is impossible from a template.
- A structured feedback control. Not a free-text box: a small set of reasons that map to the six categories above. Free-text feedback has to be read and classified by a human, which means it will not be.
- Retention that matches the support window. If tickets arrive within a fortnight and requests are retained for seven days, half the queue is undiagnosable by construction. Reconcile these two numbers deliberately, against your privacy commitments — PII in LLM logs is the constraint on the other side.
- A ticket field for the category. Added on day one, because retro-classifying a quarter of tickets is a week of work that nobody will authorise.
Triaging one ticket in five questions
Categorising is only useful if a support agent can do it in under a minute without engineering help. Five questions, in order, and the first answer that lands decides the category.
- Is there a request id? If not, the ticket cannot be diagnosed and the only honest reply says so and asks for one next time. Count these separately — the count is the business case for surfacing the id in the interface.
- Did the request fail, or succeed? A failed request is categories 4 or 6 and is answerable from the error alone. A successful request that the user considers wrong is categories 1, 2 or 3, and needs the next three questions.
- Was the task inside the documented scope? If it was not, the ticket is category 1, and the useful output is not the reply — it is a note of what the user tried, because a queue of out-of-scope attempts is the best roadmap input the feature will ever get.
- Is the output defensibly correct? Read the stored response against the stored input. If the output is right by the product’s own definition, the ticket is category 2 and the fix is a wording change at the point of use, not a model change.
- Only now is it category 3. Attach the request id to a defect, and add the input to the evaluation set the same day. A wrong answer that does not enter the evaluation set will recur.
The fifth step is the one that decides whether the queue shrinks over time. Support tickets are the highest-quality source of real failing inputs any team has, and they are usually thrown away after the reply is sent.
The changes that remove the most tickets
Ordered by how much load they remove per unit of engineering, on the basis of which categories each addresses:
| Change | Description |
|---|---|
| Narrow the input | Replace or supplement the open box with named actions covering the head intents. Removes most of category 1 outright, because the interface stops promising what the feature cannot do. |
| Say what it does before it runs | One line describing the scope and the known limits, at the point of use rather than in documentation. Addresses categories 1 and 2, and costs a sentence. |
| Validate and repair the output shape | Schema validation with a repair pass, plus a visible failure when repair does not work. Removes category 4, which is the cheapest large category to eliminate. |
| Label the output as generated | Unambiguously, every time, including in anything exported or emailed. Removes most of category 5's first question and is increasingly a regulatory expectation. |
| Show limits before they are hit | Remaining quota, current spend, and what happens at the ceiling. Category 6 tickets are almost entirely surprise, and surprise is a display problem. |
| Make abstention a first-class output | A feature that says 'I do not have this' produces far fewer tickets than one that guesses, because a confident wrong answer generates a ticket and a refusal generates a retry. |
The residual load you cannot remove
Some of this load is permanent, and planning as though it is not is how teams end up under-staffing support after an AI launch.
- Category 3 never reaches zero. A non-deterministic system will be wrong sometimes, and each instance is a real ticket with a real user behind it. Budget for it rather than treating each as an incident.
- Diagnosis is slower per ticket. Even with a request id and a stored payload, working out whether an output was wrong takes judgement that a deterministic bug does not require. Handling time goes up even when volume goes down.
- Agents need different training. A support agent who cannot tell a limitation from a defect escalates everything, which moves the load to engineering rather than reducing it. This is a training cost, and it is recurring.