LLM-as-a-Judge: Setting One Up That You Can Trust

Using a model to grade another model’s output is the only approach that scales to open-ended text. It is also the point at which your measurement device becomes a second stochastic system with opinions, and the difference between a useful judge and a number-generator is entirely in whether you validated it.

A judge is an instrument, not an oracle

Think of the judge the way a lab thinks about a thermometer. It has a reading, a bias, a precision, and a range over which it is trustworthy — and none of those are known until you check it against a reference. The reference is human labels. There is no way around this: a judge whose agreement with humans on your task is unknown produces numbers whose meaning is unknown, however many decimal places the harness prints.

The good news is that the calibration is a one-off cost of a few hundred human labels, after which the judge runs for essentially free on every subsequent evaluation. That trade is what makes judges worth the trouble.

What the published agreement figures say

The standard reference is Zheng et al., 2023, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena”. On their setup, a strong judge model agreed with human expert preferences at a rate above 80% — which the authors note is comparable to the agreement rate between two human experts on the same comparisons. That framing is the important part: the ceiling for a judge is not perfect agreement, it is human-human agreement, because the humans disagree with each other on genuinely ambiguous items.

The same paper documents the failure modes that come with it — position bias, verbosity bias, self-enhancement bias, and weakness on maths and reasoning items where the judge must itself solve the problem to grade it. So the honest summary of the literature is: a well-constructed judge on general chat quality can approach human-level agreement, and it does so while carrying systematic biases that you have to design around. It is not evidence that your judge, on your rubric, agrees with your annotators at any particular rate. That is the number you have to produce.

Designing the judge prompt

Judge prompts fail in predictable ways, and most of them are fixed by being more specific than feels necessary.

• Give criteria, not adjectives. “Rate the helpfulness” invites the judge’s own aesthetics. “Does the reply state the refund window in days? Does it avoid promising an outcome? Does it name the next step?” is gradeable. Criteria you could hand to a new employee are criteria a judge can apply.
• Supply the reference where one exists. Reference-guided grading — showing the judge a known-good answer — is the single largest improvement available for anything with a correct answer, and Zheng et al. recommend it specifically for maths and reasoning items where an ungrounded judge grades badly.
• Make it reason before it scores. Ask for a short justification, then the verdict, in that order. The ordering matters mechanically: a verdict emitted first conditions the justification into a rationalisation of it.
• Constrain the output shape. A final line of VERDICT: pass or a small JSON object. Parsing free text for a score is a source of silent grader failures that look like model regressions.
• Include the ties. If your scale has no way to express “these are equivalent” or “this item is unanswerable”, the judge will invent a distinction rather than leave the field blank, and you will read that invention as signal.

Choosing the output scale

The most common mistake is a 1-10 scale. Ten levels demand a shared notion of what separates a 6 from a 7, which does not exist even among humans, and the resulting scores cluster in a narrow band with the extremes almost unused. You get a metric with one effective bit of information and a lot of apparent precision.

The rubric-sum option deserves more use than it gets. Five yes/no criteria produce a score from 0 to 5 with far better agreement than a direct 1-5 rating, and when a model regresses you can see which criterion moved.

Calibrating against humans

The procedure, once, before the judge is trusted:

• Sample 150 to 300 items spanning the full range of quality. Include deliberate failures — a judge that has never seen a bad output in calibration is uncalibrated exactly where it matters.
• Have at least two humans label each one independently against the same written rubric the judge gets, and compute inter-rater agreement between the humans first. If they do not agree with each other, the rubric is the problem and no judge will fix it.
• Compute judge-versus-human agreement with the same statistic — Cohen’s κ for binary, not raw percentage, because raw percentage agreement is inflated when one class dominates.
• Compare judge-human agreement against human-human agreement. Roughly equal is the realistic success condition. Materially worse means the rubric is under-specified for the judge, not that the judge is stupid.
• Keep the labelled set. It is now a regression test for the judge itself, and you will need it the first time you change judge models.

Operating a judge over time

Pin the judge model to a specific version and treat changing it the way you would treat replacing a measuring instrument mid-experiment: re-run the calibration set, and be aware that historical scores are not comparable across the change. Pin the temperature at zero. Keep the judge prompt in version control next to the eval, because a judge prompt edit moves every historical number and it is the first thing to check when a metric jumps for no reason.

Finally, do not use the model under test as its own judge if you can avoid it. Self-preference is a documented and measurable effect, and the fix is cheap.