Calibration Plot Builder
Paste confidence scores and outcomes to see whether 80% confident really means right eight times in ten.
Averaged over 34 examples: when this system says it is X% sure, it is right about 18.1% away from X, weighted by how often each confidence level occurs. Worst single bin is off by 28.7%.
| bin | n | mean confidence | observed | gap | 95% interval |
|---|---|---|---|---|---|
| 0.10–0.20 | 1 | 15.0% | 0.0% | −15.0% | 0%–79% |
| 0.20–0.30 | 2 | 22.5% | 0.0% | −22.5% | 0%–66% |
| 0.30–0.40 | 2 | 32.5% | 50.0% | +17.5% | 9%–91% |
| 0.40–0.50 | 3 | 43.7% | 33.3% | −10.3% | 6%–79% |
| 0.50–0.60 | 4 | 53.8% | 25.0% | −28.7% | 5%–70% |
| 0.60–0.70 | 4 | 64.0% | 50.0% | −14.0% | 15%–85% |
| 0.70–0.80 | 5 | 74.2% | 60.0% | −14.2% | 23%–88% |
| 0.80–0.90 | 7 | 84.6% | 57.1% | −27.4% | 25%–84% |
| 0.90–1.00 | 6 | 92.7% | 83.3% | −9.3% | 44%–97% |
- Brier score — mean squared error of the probabilities, lower is better
- 0.2346
- Reliability — squared calibration gap, weighted by bin size. Lower is better
- 0.0384
- Resolution — how far the bins move from the base rate. Higher is better
- 0.0540
- Uncertainty — the base rate's own variance, which no model can beat
- 0.2500
- Residual: Brier − (reliability − resolution + uncertainty)
- 0.0002
- Maximum calibration error (worst bin)
- 28.7%
- Average stated confidence
- 66.1%
- Actual success rate
- 50.0%
- Brier score of always predicting the base rate
- 0.2500
- warnbin 0.10–0.20only 1 example — this point on the plot carries almost no information
Expected: at least 20 or so per bin before a gap is worth reading; use fewer bins or more data
- warnbin 0.20–0.30only 2 examples — this point on the plot carries almost no information
Expected: at least 20 or so per bin before a gap is worth reading; use fewer bins or more data
- warnbin 0.30–0.40only 2 examples — this point on the plot carries almost no information
Expected: at least 20 or so per bin before a gap is worth reading; use fewer bins or more data
- warnbin 0.40–0.50only 3 examples — this point on the plot carries almost no information
Expected: at least 20 or so per bin before a gap is worth reading; use fewer bins or more data
- warnbin 0.50–0.60only 4 examples — this point on the plot carries almost no information
Expected: at least 20 or so per bin before a gap is worth reading; use fewer bins or more data
- warnbin 0.60–0.70only 4 examples — this point on the plot carries almost no information
Expected: at least 20 or so per bin before a gap is worth reading; use fewer bins or more data
- noteaverage confidence is 66.1% but the actual success rate is 50.0% — systematic overconfidence of 16.1%
Expected: the two to be within a couple of points of each other on a calibrated model
Everything on this page runs in your browser. Nothing you paste is uploaded, logged, or put in the URL — only the settings above the input are, so a configured tool can be linked to.
What a calibration plot actually asks
Not "is the model accurate" — that is a different plot. The question here is narrower and more useful: when this thing says it is 80% sure, is it right 80% of the time? A model can be highly accurate and badly calibrated (it is right often, but its confidence numbers are noise), and it can be poorly accurate and perfectly calibrated (it is wrong often, and it says so). If any downstream decision reads the confidence — routing to a human above a threshold, showing or hiding an answer, deciding whether to retry — then calibration is the property you depend on, and accuracy is not.
Reading the dots
Dots below the diagonal are overconfidence: the model claimed more certainty than it earned. This is the normal failure mode for scores a language model produces about its own output, particularly when asked for a percentage in prose, and the reason "87% confident" in a chat reply should not be piped into anything. Dots above the line are underconfidence, which is rarer and less dangerous but wastes escalations.
A curve that is wrong but consistently wrong is fixable. Fit a monotonic mapping from stated to observed on held-out data — isotonic regression or Platt scaling — and the calibration error largely disappears without retraining anything. What no post-hoc fix can recover is resolution: if the model gives everything a confidence between 0.7 and 0.8, calibration can move that band to the right place, but the scores still cannot separate the cases, and the resolution term above will stay near zero.