Human Evaluation: When You Still Need People

Model judges made human evaluation cheaper, not obsolete. They moved it: instead of grading every run, humans now grade the few hundred items that decide whether the judge can be believed. That is a smaller job done more carefully, and it has its own failure modes.

Three jobs only humans can do

• Calibrating the judge. A model judge’s output is meaningless until compared with human labels on your own task and rubric. Everything else in your eval stack rests on this, and it is the job people skip.
• Deciding what good means. Whether a reply is too formal for your brand, whether a refusal was appropriate, whether an explanation would satisfy the person who asked — these are preferences of your organisation, not properties of text. No model can derive them; it can only apply them once someone has written them down.
• Discovering failure modes you have not named. Every automated grader checks for things you already thought of. A human reading a hundred outputs will find the thing nobody put in the rubric, which is the highest-value output of a human round and the reason free-text notes should always be collected alongside labels.

Guidelines that produce agreement

Disagreement between annotators is almost always the guidelines’ fault. The pattern that works:

• Decompose into independent binary questions. “Rate the quality 1-5” will not produce agreement. “Does the reply contain a factual claim not supported by the source? yes/no” will. Sum the binaries afterwards if you want a graded score.
• Give a decision order. Check the hard disqualifiers first — unsupported claim, policy violation, wrong language — and stop. Annotators who evaluate everything simultaneously weigh things differently from each other; annotators following the same decision tree do not.
• Anchor each level with a real example. Two or three per category, taken from real outputs, including at least one near-miss that is labelled the harder way with a sentence explaining why. The near-misses do all the work.
• Name the edge cases explicitly. Empty output. Output in the wrong language. A correct answer in a bad format. A refusal. Every one of these will occur, every one splits annotators who have not been told, and each takes one line to resolve.
• Keep a running decisions log. When a question comes up, the answer goes in the log and the log ships with the guidelines next round. A guideline document that does not grow is a guideline document whose ambiguities are being resolved privately and differently by each annotator.

Why raw agreement lies

Report percentage agreement and you will overstate your reliability whenever one label dominates — which, in LLM evaluation, is nearly always, because most outputs pass. Work an example. Two annotators, 100 items, binary pass/fail:

                 B: pass   B: fail
   A: pass          80         9        89
   A: fail           5         6        11
                    85        15       100

Observed agreement   p_o = (80 + 6) / 100 = 0.86
Chance agreement     p_e = (0.89 * 0.85) + (0.11 * 0.15)
                         =  0.7565 + 0.0165 = 0.7730

Cohen's kappa        k = (p_o - p_e) / (1 - p_e)
                       = (0.86 - 0.773) / (1 - 0.773)
                       = 0.087 / 0.227
                       = 0.383

Eighty-six percent agreement sounds strong. Corrected for the agreement two annotators would reach by chance given how often they each say “pass”, it is κ = 0.38 — the “fair” band in the widely-cited Landis and Koch (1977) scheme, and nowhere near good enough to certify a judge with. This is the well-known prevalence effect on kappa (Feinstein and Cicchetti, 1990): when the marginal distribution is skewed, chance agreement is high and kappa is harsh.

The practical consequence is a design instruction, not just a caveat. If your calibration sample is 90% passes, you cannot measure reliability well on it. Deliberately oversample failures — half the calibration set should be items you expect to fail — and note that this stratification means the set is for measuring agreement, not for estimating a pass rate.

Which statistic, and what target

For targets, Krippendorff’s own recommendation is the most commonly cited concrete threshold: rely on data with α ≥ 0.800, and treat 0.667 as the lowest bound at which tentative conclusions are acceptable. Landis and Koch’s bands for kappa — 0.41-0.60 moderate, 0.61-0.80 substantial, above 0.81 almost perfect — are descriptive labels rather than a standard, and it is worth saying so when you quote them.

The number that actually matters, though, is relative. Human-human agreement is the ceiling for any automated judge you build. If your annotators reach κ = 0.55 with each other, a judge reaching κ = 0.55 against them is doing as well as the task permits, and chasing 0.8 is chasing an artefact.

Running a round without wasting it

• Pilot on 20 items first. Two annotators, compute kappa, read every disagreement together, revise the guidelines. Only then label the full set. Skipping the pilot is how a 500-item round gets thrown away.
• Randomise order and blind the source. Annotators who can see which system produced an output will label it accordingly, and this is not a matter of good faith — it is the same position and provenance effect model judges show.
• Overlap 10-20% for ongoing agreement. Once past the pilot, keeping a shared subset lets you detect drift, and annotator drift over a long round is real.
• Cap sessions. Agreement degrades with fatigue. Ninety minutes or a hundred items, whichever comes first, is a common working limit.
• Keep the disagreements. Items two careful people labelled differently are the most informative items you own: they mark the boundary of your own definition of quality, and they belong in the guidelines as anchors for the next round.

On who does the labelling: for calibration rounds, prefer people who know the domain over people who are cheap. A crowd worker can tell you whether a summary is fluent; only somebody who understands your refund policy can tell you whether a reply about it was correct, and correctness is the axis your judge most needs to learn. Two informed annotators for a day usually produce a more useful calibration set than fifty uninformed ones for a week, and the resulting agreement statistic actually means something.