User Feedback as an Eval Signal

Thumbs up and thumbs down are the cheapest evaluation signal there is, which is why they end up on a dashboard as a quality percentage within a week of launch. They are worth collecting. They are not worth reading as a level, and the reasons are structural rather than fixable by asking nicer.

Why the thumbs rate is not a quality rate

• Response rates are tiny and non-random. Only a small fraction of users ever click, and the ones who do are disproportionately those with a strong reaction. Your feedback sample is drawn from the tails, so the mean of it is not the mean of anything.
• The denominator is ambiguous. Thumbs-down over thumbs-total, over responses shown, or over sessions? These three numbers move independently — a UI change that makes the buttons more visible moves two of them and not the third — and teams switch between them without noticing.
• Users rate the outcome, not the response. A correct answer that says “no, that is not refundable” collects thumbs-down. A wrong answer that says yes collects thumbs up. On any system that delivers bad news, feedback is partially an inverted quality signal, and that inversion is concentrated in exactly the segments you most need to be right in.
• Placement effects dominate small changes. Button position, whether a comment box appears, whether feedback is requested after a delay. A two-point movement in a thumbs rate is more likely to be a front-end deploy than a model change, which makes the metric useless for exactly the comparison people want it for.

The consequence is not “ignore feedback”. It is that feedback is a relative signal — comparable across time only when the interface is frozen, and comparable across segments only when the segments have similar populations. Never a level.

Implicit signals carry more

Behaviour is denser than opinion. Every user produces it, and it is much harder to distort by moving a button.

The edit-distance signal deserves particular attention because it is continuous, universal within its product category, and almost never instrumented. It converts every single use of the product into a graded example, at zero cost to the user, and it is far more sensitive than a binary rating. Normalise it — edits per hundred characters rather than raw distance — so that long drafts do not dominate, and record it per response rather than per session so it joins cleanly to everything else you log.

Establishing correlation, not assuming it

Whether any of these signals tracks quality in your product is an empirical question about your users, and there is no general answer to import. It is also cheap to answer once, and the answer is what licenses everything you do with the signal afterwards.

• Sample 400 production interactions, stratified deliberately: 100 with thumbs-down, 100 with thumbs-up, 100 with a retry and no explicit rating, 100 with neither. Stratifying is essential — with natural rates you would collect a handful of negatives.
• Grade all 400 blind against your existing rubric, by humans or by a judge you have already calibrated. The graders must not see the feedback signal, or the study measures nothing.
• For each signal, build the 2×2 against the graded verdict and compute precision and recall. “Of retried responses, what proportion were genuinely bad” is the number that determines whether retry is usable as a triage filter.
• Compute the association properly: point-biserial or Spearman correlation for a graded score, or Cohen’s κ for binary against binary. Report the confidence interval; with 400 stratified items it will be wider than you expect.
• Re-run it after any interface change to the feedback mechanism, and annually regardless. The relationship is a property of your user population, and populations move.

The result you get is the licence. A signal with high precision and low recall is a good triage filter and a bad metric. A signal with a weak correlation is a signal to stop putting on the dashboard, and finding that out costs one afternoon rather than a year of decisions made on a number that meant nothing.

Reporting feedback honestly

Three rules that prevent most of the damage. Always report the response rate next to the feedback rate — a thumbs-down rate of 4% at a 2% response rate is 0.08% of traffic and should never be printed without that context. Always put an interval on it: with small counts the Wilson interval is the right one, and “3 of 41” has an interval wide enough to make the point on its own.

And segment before you trend. An aggregate feedback rate moves when the traffic mix moves even if quality is identical in every segment — onboarding a large customer whose users are harder to please drops the aggregate with nothing having changed for anyone. A trend line on a mixed population is a trend line about the mix.

The use that always works

Independent of everything above, negative feedback is the best-targeted queue of candidate eval items you will ever have. It costs nothing to collect, it is already filtered by a human who was there, and each item arrives with the context of what the user was trying to do.

So treat feedback as a sampler rather than a metric. Grade the flagged interactions properly, promote the genuine failures into the golden set with their provenance recorded, and let the graded score — not the thumbs rate — be the number on the dashboard. That way the cheap signal does the job it is good at, and the expensive signal does the job it is good at.