Inter-Rater Agreement Calculator
Paste two columns of labels and get Cohen's kappa, weighted kappa, the confusion matrix and the reason your raters disagree.
20 items, 2 categories. Weak agreement. The rubric is probably ambiguous.
- Observed agreement (p₀)
- 80.0%
- Agreement expected by chance (pₑ)
- 54.0%
- Unweighted Cohen's kappa
- 0.565
- 95% interval for unweighted kappa
- 0.196 to 0.934
- PABAK (prevalence-and-bias-adjusted)
- 0.600
- Bias index — how differently the two raters used the categories
- 0.100
| A ↓ / B → | bad | good | total |
|---|---|---|---|
| bad | 5 | 1 | 6 |
| good | 3 | 11 | 14 |
The diagonal is agreement. The largest off-diagonal cell is the confusion worth rewriting the rubric for.
- warn20 rowsthe confidence interval below uses a large-sample approximation that is unreliable at this size
Expected: roughly 30 rows before the interval means much, and more if any category is rare
# are skipped. The weighted interval is not reported because the variance formula above is the unweighted one; weighted kappa is shown as a point estimate only.Everything on this page runs in your browser. Nothing you paste is uploaded, logged, or put in the URL — only the settings above the input are, so a configured tool can be linked to.
Why raw agreement flatters you
Two raters labelling spam with a 95% base rate will agree about 90% of the time by pressing "spam" without reading. Kappa is the correction: it subtracts the agreement you would expect from chance alone and rescales what is left, so 0 means "no better than coin flips with your marginal rates" and 1 means perfect. This is exactly the check an LLM-judge needs before you trust it — a judge that agrees with your human 92% of the time on a set that is 90% pass has told you almost nothing.
The paradox, and what to do about it
On heavily skewed data, kappa can fall to near zero while observed agreement stays above 90%. That is not a bug, and it is not a reason to drop kappa; it is a real statement that on this distribution the raters have not demonstrated skill. The practical fix is to look at PABAK and the bias index alongside it, and better still to re-balance the sample you use for the agreement check so the rare category is not rare. If you only ever validate a judge on data that is 95% one class, you will never learn whether it can find the other 5%, which is usually the entire reason you built it.
Whatever the number, read the largest off-diagonal cell in the matrix above. Agreement statistics tell you there is a problem; that cell tells you which pair of categories your rubric fails to separate.