Detecting Quality Regressions in Production

Quality regressions do not show up in your error rate, your latency or your status page. They show up in a support queue three weeks later. The only defence is a set of measurable proxies — and honesty about which of them your traffic is large enough to detect a change in.

The 200 that is wrong

Everything a conventional monitoring stack watches is about whether the request completed. An LLM feature can complete every request, within latency budget, at normal cost, and be materially worse than it was last week: vaguer, less grounded, refusing things it used to answer, quietly dropping a field from its JSON.

The causes are boringly ordinary — a prompt edit, a retrieval index rebuild, a model version change, a truncation limit reached because documents got longer, a tool description reworded. What they have in common is that none of them produces an error, so none of them produces a signal unless you built one.

Four tiers of proxy signal

Order them by how cheap they are to compute and how directly they relate to what the user experiences. The cheap ones are noisy proxies; the direct ones are expensive and lagging. You want several from different tiers, because a regression that moves none of them is rare and a regression that moves all of them is unambiguous.

Tier 1 — free, computed from the log you already have

• Schema validity rate. The share of outputs that parse and validate against the expected structure. The single best signal available if your output is structured, because it is exact, has no judgement in it, and moves immediately.
• Truncation rate. finish_reason = 'length'. Rises whenever prompts grow or the model becomes more verbose.
• Output length distribution. Not the mean — the distribution. A model that has started padding or started clipping shifts the shape well before anyone complains. Compare distributions with a two-sample Kolmogorov–Smirnov test or a population stability index rather than eyeballing an average.
• Retry and tool-error rate. Agents that begin failing tool calls are producing malformed arguments, which is a quality regression wearing an infrastructure costume.

Tier 2 — cheap heuristics over the output text

• Refusal rate. Detected with a small classifier or a curated pattern list. Worth building deliberately: refusals are the most common way a model change becomes user-visible, and they are invisible in every other metric.
• Citation or grounding rate. For RAG, the share of answers containing at least one reference that resolves to a retrieved document id. Exact, cheap, and directly tied to the thing RAG is for.
• Empty-ish answers. Below a length floor, or matching hedging templates. Trivial to compute, and a step change here is almost never benign.

Tier 3 — user behaviour

• Regeneration rate — how often a user asks again immediately. The closest thing to a free label you will get.
• Edit distance before accept, where your product lets a user accept or modify a suggestion. It degrades gracefully and correlates with usefulness better than a thumbs control does.
• Explicit feedback, with the caveat that it is sparse, heavily biased toward the annoyed, and slow. Useful as a corroborator, dangerous as a trigger.

Tier 4 — a graded sample

A model-based or human judge scoring a random sample of production traffic against a rubric. It measures what you actually care about and it is the slowest, most expensive and noisiest of the four. Treat it as the confirmation step, not the detector.

What your traffic volume can detect

This is the part that is usually skipped, and skipping it is why teams argue about whether a chart moved. For a rate-type signal, the sample size needed to detect an absolute change of δ in a baseline rate p, at the conventional 5% significance and 80% power, is approximately:

n per group  ≈  (z(α/2) + z(β))² · 2 · p · (1 − p) / δ²
             ≈  (1.96 + 0.84)²  · 2 · p · (1 − p) / δ²
             ≈  7.84 · 2p(1 − p) / δ²

Worked, for a baseline schema-failure rate p = 2%:
  detect δ = 1.0 pp (2% → 3%)   →  n ≈ 3,100 per group
  detect δ = 0.5 pp (2% → 2.5%) →  n ≈ 12,300 per group
  detect δ = 0.2 pp             →  n ≈ 76,900 per group

Worked, for a baseline refusal rate p = 10%:
  detect δ = 2.0 pp             →  n ≈ 3,500 per group
  detect δ = 1.0 pp             →  n ≈ 14,100 per group

Read that as a capability statement about your monitoring. At 5,000 requests a day on a feature, you can see a schema-failure rate double within a day and you cannot see a half-point move inside a week. So do not build an alert on the half-point move; build a weekly review for it, and put the alert where the arithmetic says it will work.

The same calculation is what tells you how long a canary stage has to run before its comparison means anything. A 1% canary on that same feature accumulates 50 requests a day, which will not detect anything at all — which is an argument for a larger first stage, not for a longer wait.

The sampled judge and its error bars

When you do grade a sample, report the interval, not the point. A rubric score of 4.1 from 100 graded examples is not different from 4.3 from another 100, and treating it as different is how a team spends a week chasing noise. For a proportion, the normal-approximation interval half-width is 1.96 · sqrt(p(1−p)/n) — at p = 0.9 and n = 100 that is about ±5.9 points, which is wider than most of the differences people act on.

Two practices make judged samples more useful. Stratify the sample so rare-but-important request types are represented, rather than sampling uniformly and grading a hundred easy questions. And keep a frozen reference set graded by the same judge version, so that a change in the score cannot be caused by a change in the judge — judges are models too, and they get silently updated exactly like the model under test.

Wiring it up

The tier-1 and tier-2 signals should be computed at write time, not at query time, and stored as columns on the request row. That makes them groupable by prompt_version, served_model and release, which is what turns “quality dropped” into a named cause.

alter table llm_request
  add column schema_valid   boolean,
  add column refused        boolean,
  add column citation_count smallint,
  add column regenerated    boolean;   -- backfilled from the UI event

-- The daily quality board, split by everything that could have caused it.
select started_at::date                                        as day,
       served_model, prompt_version, release,
       count(*)                                                as n,
       round(avg((not schema_valid)::int)::numeric, 4)         as schema_fail,
       round(avg(refused::int)::numeric, 4)                    as refusal,
       round(avg((finish_reason = 'length')::int)::numeric, 4) as truncated,
       round(avg((citation_count = 0)::int)::numeric, 4)       as ungrounded,
       round(avg(regenerated::int)::numeric, 4)                as regen
from llm_request
where environment = 'prod' and feature = $1
  and started_at >= now() - interval '30 days'
group by 1, 2, 3, 4
having count(*) >= 500          -- below this the rates are noise
order by day desc;

The having clause is doing real work. Groups too small to support a rate should not appear on the board at all, because a row reading “40% refusal” from five requests will be believed by somebody.

Two final habits. Recompute the tier-2 signals if you change how they are defined, and version the definition — a refusal detector that was quietly improved makes last month’s refusal rate incomparable with this month’s, which is the same silent-ruler problem as an unpinned judge. And review the board on a fixed cadence even when nothing has alerted, because the regressions that matter most are the slow ones that never cross a threshold in any single window.