Rubrics Two Judges Agree On

A rubric is good when two judges applying it to the same answers reach the same scores. That is measurable, it is the only property of a rubric worth optimising, and it is measurable before you have any human labels at all.

One criterion, written out

<criterion name="faithfulness">
Does the answer state anything <source> does not support?

5  Every claim in the answer is traceable to a span in <source>. No added
   qualifiers, no added causes, no added consequences.
3  Exactly one claim is not supported by <source>, and it would not change
   what a reader does next.
1  A claim is not supported by <source> and it would change what a reader
   does next, OR the answer contains a figure, date or name that does not
   appear in <source>.

Score 5, 3 or 1 only. If you want to score 4, decide which of 5 and 3 the
answer is nearer and say why in "note". If you want to score 2, the answer
is a 1.

Return: {"score": 5|3|1,
         "quote": "<the span from <source> that decides the score, verbatim>",
         "note": "<one sentence, only if you nearly chose the other level>"}

Judge faithfulness only. Do not consider style, length, helpfulness, tone, or
whether the answer is true in the world. Another criterion covers each of
those.
</criterion>

Three properties make this usable by two different judges. The levels are defined by an observable feature of the answer, not by an adverb. The boundary between 3 and 1 is a consequence test — would it change what a reader does — which is a question two people answer the same way far more often than “is this a serious problem”. And the quote requirement forces the judge to locate the deciding evidence, which is the same device that makes extraction and code review checkable.

Score one criterion per call. A single call scoring five criteria anchors every later score on the first one — the judge that has just given a 1 for faithfulness is reluctant to give a 5 for helpfulness to the same answer. Separate calls cost more and remove the effect entirely, and for a rubric you are still developing that is the right trade.

Why only odd levels

A 1-to-5 scale with all five levels available concentrates disagreement on 2 and 4, and the mechanism is not mysterious: those levels are defined by their neighbours rather than by anything observable. “Slightly worse than the 5” is a comparison to a judgement, so any variation in where a judge puts 5 propagates into 4.

Removing them costs less information than it appears to, because the decision downstream is almost always a threshold — does this pass, does this need review, does this block the release. A three-level scale answers that question directly, and it makes the two judges agree far more often simply by removing the two values they were arguing about.

The instruction for what to do when you want a 4 is what stops the model splitting the difference anyway. Without it, models return 4 and your parser either rejects the response or silently rounds. Sending the near-miss to note keeps the information without keeping the level, and reading a week of notes tells you where the anchors need rewriting.

Absolute scoring against anchors is not always the right instrument. If you only need to know which of two systems is better, pairwise comparison is more stable — the trade-off is set out in pairwise comparison versus absolute scoring. Use anchored scores when you need a number that survives the other system being replaced.

Ordering controls

Judges are sensitive to the order things appear in. Four controls, in descending order of how much they matter:

1. Swap and repeat, for any comparison. Score A-then-B and B-then-A in two separate calls. Keep the verdict only where the two agree; the disagreements are your position-bias rate, and you should know it rather than assume it away. The effect and how to quantify it are covered in judge bias.
2. Randomise or isolate criterion order. One criterion per call is the clean fix. If you must batch, shuffle the order per item so the anchoring effect is noise rather than a consistent bias.
3. Strip identity. The judge must not see which model produced an answer, which is the incumbent, or which the author prefers. This includes indirect leaks: a distinctive refusal phrasing or a formatting habit identifies a model as reliably as a label.
4. Match lengths in the eval set, do not instruct. Telling a judge that length is not a criterion helps less than building an eval set whose answers are of comparable length. The instruction addresses the judge’s stated reasoning; the set construction addresses the score.

Run judges at temperature 0. Not because it removes bias — it does not — but because it removes one source of variance, and you cannot diagnose the others while sampling noise is in the way.

Measuring agreement

Raw agreement is misleading when scores are unbalanced: if 85% of answers score 5, two judges who always say 5 agree 85% of the time and have told you nothing. Cohen’s kappa corrects for agreement expected by chance, and it is short enough to keep in your own repository.

from collections import Counter

def cohens_kappa(a: list, b: list) -> float:
    """Chance-corrected agreement between two raters over the same items.

    1.0  identical.   0.0  no better than chance.   negative  worse than chance.
    """
    assert len(a) == len(b) and a, "need two equal, non-empty score lists"
    n = len(a)
    labels = set(a) | set(b)
    po = sum(1 for x, y in zip(a, b) if x == y) / n
    ca, cb = Counter(a), Counter(b)
    pe = sum((ca[l] / n) * (cb[l] / n) for l in labels)
    return 1.0 if pe == 1.0 else (po - pe) / (1.0 - pe)


def per_criterion(scores_a: dict, scores_b: dict) -> dict:
    """scores_a[criterion] = [score per item]. Same items, same order."""
    return {c: round(cohens_kappa(scores_a[c], scores_b[c]), 3)
            for c in scores_a if c in scores_b}

Reading the result. The conventional bands — slight, fair, moderate, substantial — are a rule of thumb from the human-annotation literature rather than a law, so treat them as orientation. What matters practically is the comparison between criteria and against the ceiling below.

• Compute it per criterion, never overall. An overall figure hides the one criterion that is broken, and there is nearly always exactly one.
• Fix the lowest-scoring criterion first, and fix the anchors. Pull the twenty items where the judges disagreed and read them. In most cases they cluster on a single distinction the anchors do not make, and one added sentence resolves them.
• Two judges can be two models, or one model in two configurations. Different models is the stronger test. Different prompt orderings of the same model is a cheaper one that still finds anchor ambiguity.

The self-agreement ceiling

Before comparing two judges, measure one judge against itself: run the same judge twice over the same items, in fresh calls, at temperature 0, and compute kappa between the two runs.

That number is your ceiling. If a single judge scoring the same answers twice agrees with itself at 0.7, no rubric edit will get two different judges above 0.7 — the instability is in the criterion, not in the disagreement between judges. This single control saves the most common wasted week in evaluation work: tuning a judge prompt to close a gap that the rubric itself cannot close.

1. Run judge X over the item set. Save the scores.
2. Run judge X again, fresh context, same temperature, same order. Save the scores.
3. cohens_kappa(run1, run2) per criterion. This is the ceiling.
4. Run judge Y. Compare X-run-1 against Y. If that number is close to the ceiling, your rubric is as good as it can be and remaining disagreement is judge variance.
5. If the ceiling itself is low for a criterion, rewrite that criterion’s anchors before touching anything else.

When it stops working

• Kappa falls on one criterion after a prompt edit. Recompute per criterion on every rubric change; it is a two-minute job with the harness above and it localises the regression to the criterion you touched.
• Scores pile onto one level. If 90% of items score 5, the criterion has stopped discriminating — either the system genuinely improved past it or the anchors have drifted. Check with an item you know is a 1.
• note fills up on the same boundary repeatedly. The most direct signal there is: the judges keep nearly choosing the other level at the same place, and that place is where the anchor needs a sentence.
• The self-agreement ceiling drops. Usually a model change rather than a rubric change. Re-measure the ceiling before concluding anything about the rubric.