Feedback Widgets That Produce Usable Data
6 min read · updated August 3, 2026
A thumbs-down tells you that somebody, whose identity within your user base is not random, disliked something, for one of about nine reasons, at some point in a pipeline with about six stages. Every one of those ambiguities is fixable, and the fixes are mostly not about the widget.
What a thumb actually tells you
Be precise about the information content, because the widget is usually defended on the grounds that some signal beats none. A thumbs-down establishes: an output was produced, a person looked at it, and their reaction was negative enough to exceed the effort of clicking.
It does not establish which of these happened: the retrieval missed the document, the retrieved document was wrong, the model ignored the context, the model was factually wrong, the format was unusable, the answer was fine but slow, the answer was fine but declined something adjacent, the answer answered a different question than the one meant, or the user was annoyed about something else entirely. Nine causes, one bit. And the bit does not say which — which is the reason the ratings pile up in a dashboard that nobody has ever acted on.
Two structural problems
Selection bias, and why the rate is not a quality metric
The population that rates is not the population that uses. Rating requires a reaction strong enough to overcome the effort of clicking, and the threshold is not symmetric — a mildly good answer produces no click while a mildly bad one often does, because irritation is more activating than satisfaction.
The consequence is specific: your thumbs-up rate cannot be read as a quality estimate, and its changes confound quality with anything that alters who bothers to click — placement, a redesign, a growth campaign that brings in less invested users. Treat the rate as an uncalibrated instrument. Its absolute value means nothing; even its direction over time is only interpretable if nothing about the population or the widget moved.
The missing counterfactual
A rating says the output was bad. It never says what good would have been. That is the piece you need to fix anything, and it is exactly the piece the interaction does not collect — which is why teams end up reading individual traces by hand anyway.
The only cheap source of a counterfactual is the user’s own correction. If they edited the output before using it, the edited version is the counterfactual, produced for free by someone who knew what they wanted. That observation is most of the last section of this page.
The payload is worth more than the label
A rating is only actionable if you can reconstruct what was rated. That means capturing the trace at rating time, not the label alone, and it is the difference between a dashboard and a debugging tool.
type FeedbackEvent = {
// Identity of the thing being rated
requestId: string; // joins to the provider-side record
traceId: string; // joins to the whole pipeline run
turnId: string;
// What produced it — the variables you might change
model: string;
promptVersion: string; // useless without this
temperature: number;
retrievedDocIds: string[]; // empty array is itself a finding
toolCalls: string[];
// What it cost, so you can weigh a fix
ttftMs: number;
totalMs: number;
costMicros: number;
// The rating
rating: "up" | "down";
reason?: FeedbackReason; // see below
note?: string;
// The counterfactual, if it exists
editedText?: string; // what the user changed it to
editDistance?: number;
};The field that pays for the whole schema is promptVersion. Without it, feedback collected across a prompt change is a single undifferentiated pool and you cannot tell whether the change helped. With it, every rating is attached to a specific configuration and the pool becomes a comparison. See versioning prompts as artefacts and what a trace needs to contain.
Categories that route
If you ask one follow-up question after a thumbs-down, the categories should map onto stages of your pipeline, not onto adjectives. The test for a good category set: each option, if selected often, points at a different team or a different fix.
| Option shown | Description |
|---|---|
| Facts are wrong | Points at grounding — retrieval quality, or generation drifting from the context. Actionable by the RAG pipeline. |
| Missed what I asked | Points at instruction following or an ambiguous request. Actionable by the prompt, and sometimes by the interface that collected the request. |
| Wrong format or length | Points at the output contract. Almost always the cheapest category to fix, and often the largest. |
| Refused something reasonable | Points at safety configuration or a false refusal. Needs to be tracked separately because the fix is a policy change, not a quality change. |
| Too slow | Not a quality complaint at all. Routing it into the quality pool is how latency problems get misdiagnosed as model problems for a quarter. |
| Something else | With a free-text box. This is where the categories you have not thought of arrive; read it monthly and expect to add a row. |
One follow-up, not a form. Every additional field cuts response rate, and the second question is worth much less than the first, because the first already routed the issue.
The signals you already have
Implicit signals do not suffer from the same selection problem: they are emitted by everyone who uses the feature, not by the subset moved to rate it. They are also not opinions — they are what the user did, which is usually the more relevant question.
- Edit distance between generated and shipped. The strongest free signal available. If a user accepted the output and then changed 40% of it, the output was 60% right and you have the corrected version. Requires that you can see the final text, which is true in an editor and false in a copy-to-clipboard flow.
- Accepted, discarded, or ignored. Three outcomes, all observable, none requiring a click from the user. Distinguish discarded (an explicit reject) from ignored (they moved on), because the second is often a latency signal in disguise.
- Regeneration. A user who regenerates has told you the first answer was inadequate, more reliably than a thumbs-down, because it cost them a wait. Regenerate rate is a quality metric and it needs no widget at all.
- Abandonment mid-stream. Navigating away while text is still arriving is a latency complaint, not a quality one — they never saw the answer. Keep it out of the quality pool.
- Immediate rephrase. A follow-up message within a few seconds that restates the same request is a strong signal the first attempt missed, and it comes with the user’s own better phrasing attached.
None of this makes the thumb useless. It makes it a routing device: cheap to click, useful for flagging traces worth a human reading, and worth almost nothing in aggregate. Build it, keep the payload, and get your quality signal from what people do.