Skip to content
Multigrid

Evaluation, benchmarks & LLM-as-judge

How to find out whether a model is good at your task, when the public numbers stop being evidence, and how to grade at scale without lying to yourself.

Evaluation is the part of building on language models that everyone agrees is essential and almost nobody does properly. The reason is structural: the output is text, text has no obvious right answer, and the system that produces it is stochastic, so every instinct carried over from testing deterministic software gives the wrong result. A test that passes 4 times out of 5 is not a flaky test to be retried; it is a measurement with a confidence interval.

These pages take that seriously. They cover what public benchmarks actually contain and where they stop being predictive, how to build the small private eval set that replaces them, how to use a model as a grader without inheriting its biases, and the statistics that decide whether a change you can see in a spreadsheet is a change at all. Where a page makes a numerical claim, the claim is either attributed to published work or shipped as a script you run on your own system.

How to Evaluate an LLM for Your Own Use Case

A construction procedure for a fifty-example private eval set, and the arithmetic that tells you which decisions fifty examples can actually settle.

5 min read

Why Public Benchmarks Don't Predict Your Results

The mechanisms that break the link between leaderboard position and performance on your task, plus the rank-correlation script that tells you how bad the break is for you.

6 min read

MMLU, GPQA, SWE-bench: What Each Actually Measures

What is inside the benchmarks whose names appear on every model card, and one paragraph each on the question that benchmark cannot answer.

6 min read

Benchmark Contamination: When Models Have Seen the Test

How test data leaks into training sets, the published methods for detecting it through a closed API, and what to do about it in your own eval.

5 min read

LLM-as-a-Judge: Setting One Up That You Can Trust

Judge prompt design, the published human-agreement figures, and the validation step that has to happen before a judge's score is allowed to mean anything.

5 min read

Judge Bias: Length, Position and Self-Preference

The three documented biases of model judges, the papers that demonstrated each, and a harness that measures all three on your own judge.

6 min read

Pairwise Comparison vs Absolute Scoring

Why comparison is a more reliable judgement than rating, what it costs in judge calls, and the cases where absolute scoring is the only thing that works.

5 min read

Golden Datasets: Building and Maintaining One

A curation workflow with explicit provenance, item lifecycle states and review rules, designed so the set survives the departure of whoever started it.

5 min read

Regression Testing for Prompts and Models

A two-tier CI gate for a stochastic system: hard contract assertions that must never fail, and a statistical non-inferiority gate for quality.

6 min read

Statistical Significance With Non-Deterministic Models

The right test for paired pass/fail evaluation, a worked sample-size calculation, and how to split your budget between more items and more samples per item.

7 min read

Human Evaluation: When You Still Need People

Where human labels remain the only ground truth, how to write guidelines that produce agreement, and which agreement statistic to report with what target.

5 min read

Task-Specific Metrics: BLEU, ROUGE, and Their Limits

What the classic n-gram metrics compute, the reporting traps that make published scores incomparable, and the cases where they still beat a model judge.

4 min read

Evaluating Summarisation Without Reference Summaries

A reference-free evaluation built from claim decomposition and entailment for faithfulness, plus source-driven salience checks for coverage.

5 min read

Evaluating Code Generation Beyond pass@k

What pass@k actually estimates and why it flatters a system, plus the runtime, security and maintainability dimensions that decide whether generated code is usable.

5 min read

Red-Teaming Your Own Application

An attack playbook you can run in an afternoon, organised by published attack family, with the scoring that separates an embarrassing output from an actual breach.

5 min read

Evaluating Cost and Latency Alongside Quality

How to evaluate on three axes at once using a Pareto front, and how to collapse it to a decision when you finally have to pick one.

5 min read

Online Evaluation: Grading Production Traffic

How to grade live requests continuously, why the sample size follows from the precision you need rather than from a percentage of traffic, and how to keep a biased sample unbiased.

5 min read

User Feedback as an Eval Signal

Why a thumbs-down rate is not a quality metric, the implicit signals that carry more information, and the procedure for establishing whether your feedback correlates with quality at all.

4 min read

Eval Frameworks Compared

The families of evaluation tooling, the four axes on which each one locks you in, and the exit test to run before you commit a year of results to any of them.

4 min read

Building an Eval Harness in 200 Lines

A complete dependency-free eval harness with caching, k-sample runs, pluggable graders, bootstrap intervals and a paired CI gate — with the code on the page.

8 min read

Other topics