Calibration: Making 0.8 Mean 80%
11 min read · updated August 4, 2026
A model that outputs 0.8 is calibrated if, among all the rows it scored near 0.8, about 80% turned out positive. Most models are not, and the moment anyone multiplies a score by a monetary value, the miscalibration becomes an arithmetic error in the business case. This page diagnoses it on 1,000 predictions and fixes it, with every number shown.
When the number has to mean something
Calibration is irrelevant to a pure ranking problem. If all you do is sort leads and work down the list, the scores can be any monotone transform of the truth and the list is identical. AUC will not change when you recalibrate, at all.
It becomes essential the moment a score is used as a number rather than as an order:
- Expected value. “This customer has a 0.8 probability of a £200 renewal, so they are worth £160” is only true if 0.8 means 0.8.
- Cost-based thresholds. Deriving a threshold from a cost matrix produces a probability, and comparing an uncalibrated score against it compares two different scales.
- Combining models. Averaging or multiplying scores from two models that are miscalibrated in different directions produces something with no interpretation at all.
- Reporting to a human. An analyst told “72% likely” will act on it as a probability, and is entitled to.
Gradient-boosted models and support vector machines are usually over-confident at both ends, because the loss keeps rewarding pushing confident rows further out. Naive Bayes is badly miscalibrated by construction. Random forests are typically under-confident at the extremes, since averaging many trees pulls predictions towards the middle. Logistic regression on well-specified features is often nearly calibrated already, which is one of the quiet reasons it survives.
The reliability table
Take 1,000 held-out predictions from a boosted classifier — a calibration set the model never saw — and bucket them by predicted probability. The numbers below are invented so the arithmetic is checkable; the shape is the one over-confident models produce.
bucket mean pred n positives observed rate pred - obs
0.00-0.10 0.05 300 44 0.147 -0.097
0.10-0.20 0.15 150 39 0.260 -0.110
0.20-0.30 0.25 100 34 0.340 -0.090
0.30-0.40 0.35 80 33 0.413 -0.063
0.40-0.50 0.45 70 33 0.471 -0.021
0.50-0.60 0.55 70 37 0.529 +0.021
0.60-0.70 0.65 60 36 0.600 +0.050
0.70-0.80 0.75 60 40 0.667 +0.083
0.80-0.90 0.85 60 44 0.733 +0.117
0.90-1.00 0.95 50 43 0.860 +0.090
----- -----
1000 383 base rate = 0.383
mean predicted probability = 343/1000 = 0.343Read the last column. At the low end the model is under-confident: it says 5% and delivers 15%. At the high end it is over-confident: it says 95% and delivers 86%. The predictions are squashed outward from the middle — too sharp, in both directions — which is exactly the signature of a model trained to minimise log loss without a calibration step.
Note also that the model is ranking perfectly well. The observed rate rises monotonically down the table, so its AUC is fine and no ranking metric will report a problem. Calibration is a separate axis and needs a separate diagnostic.
Expected calibration error, summed
ECE is the average gap between prediction and outcome, weighted by how many predictions fell in each bucket. One line per bucket:
ECE = sum over buckets of (n_b / N) * |pred_b - obs_b|
bucket weight |gap| contribution
0.05 0.300 0.097 0.0291
0.15 0.150 0.110 0.0165
0.25 0.100 0.090 0.0090
0.35 0.080 0.063 0.0050
0.45 0.070 0.021 0.0015
0.55 0.070 0.021 0.0015
0.65 0.060 0.050 0.0030
0.75 0.060 0.083 0.0050
0.85 0.060 0.117 0.0070
0.95 0.050 0.090 0.0045
-------
ECE = 0.08218.2 percentage points. On a portfolio where each point of probability is worth £2 of expected value, that is £16 of error per decision, systematically in one direction at each end of the score range.
The Brier score — mean squared error between prediction and outcome — is the other number usually quoted. It is a proper scoring rule and a good overall summary, but it mixes calibration with discrimination, so it is a poor diagnostic: a model can improve its Brier score by getting better at ranking while its calibration gets worse. Use Brier to compare models and ECE with a reliability table to diagnose one.
Platt scaling, worked
Platt scaling fits a one-dimensional logistic regression on the model’s log-odds: two parameters, a slope and an intercept. Fitting it on a held-out calibration set for the table above gives roughly a = 0.60 and b = 0.06.
z = ln( p / (1 - p) ) raw log-odds z' = a * z + b a = 0.60, b = 0.06 p' = 1 / (1 + exp(-z')) corrected probability raw p = 0.95: z = ln(0.95/0.05) = ln(19) = 2.944 z' = 0.60(2.944) + 0.06 = 1.827 p' = 1/(1 + e^-1.827) = 0.861 observed was 0.860 raw p = 0.05: z = ln(0.05/0.95) = -2.944 z' = 0.60(-2.944) + 0.06 = -1.707 p' = 1/(1 + e^1.707) = 0.154 observed was 0.147 raw p = 0.25: z = ln(0.25/0.75) = -1.099 z' = 0.60(-1.099) + 0.06 = -0.599 p' = 1/(1 + e^0.599) = 0.355 observed was 0.340 Recomputing ECE over all ten buckets with the corrected values: ECE 0.082 -> 0.010
The slope a is the interpretable parameter. Below 1 it pulls predictions towards the middle, which is the correction for an over-confident model; above 1 it pushes them outward. A slope of 0.60 says the model’s log-odds were roughly 1.7 times too large.
Platt scaling is the right default when the calibration set is small — under a few thousand rows — because two parameters cannot overfit much. Its limitation is that it can only apply a sigmoid-shaped correction, so a model whose miscalibration is not that shape will be improved and not fixed.
Isotonic regression, and when to prefer it
Isotonic regression fits any non-decreasing step function from raw score to calibrated probability. It makes no assumption about shape, so it can correct arbitrary miscalibration — and with that flexibility it will happily fit noise on a small calibration set, producing flat regions and a step function that generalises badly.
| Choose | Description |
|---|---|
| Platt (sigmoid) | Calibration set under roughly 1,000–5,000 rows, or miscalibration that looks like a smooth squash — the pattern in the table above. Two parameters, very stable, cannot produce a pathological curve. |
| isotonic | Large calibration set, tens of thousands of rows, or miscalibration with a shape a sigmoid cannot reach — a flat region in the middle, a kink at a decision boundary. Set out_of_bounds='clip' or scores outside the fitted range raise. |
| neither | The model is used only for ranking, or the calibration set is a few hundred rows. Fitting a correction on 300 rows adds variance and buys nothing you can measure. |
Both must be fitted on data the model did not train on. Calibrating on the training set is leakage in its plainest form: the model is over-confident on training rows in a way that does not transfer, and the correction learned there is wrong everywhere else.
The code
import numpy as np
from sklearn.calibration import calibration_curve
from sklearn.isotonic import IsotonicRegression
from sklearn.linear_model import LogisticRegression
def ece(y_true, p, n_bins=10):
"""Expected calibration error with equal-width bins."""
edges = np.linspace(0.0, 1.0, n_bins + 1)
idx = np.clip(np.digitize(p, edges[1:-1]), 0, n_bins - 1)
total = 0.0
for b in range(n_bins):
m = idx == b
if m.sum() == 0:
continue
total += (m.sum() / len(p)) * abs(p[m].mean() - y_true[m].mean())
return total
# 1. hold out a calibration set the model never trained on
p_cal = model.predict_proba(X_cal)[:, 1]
p_test = model.predict_proba(X_test)[:, 1]
print("raw ECE:", ece(y_test, p_test))
# 2a. Platt: logistic regression on the raw log-odds
eps = 1e-6
z_cal = np.log(np.clip(p_cal, eps, 1 - eps) / (1 - np.clip(p_cal, eps, 1 - eps)))
platt = LogisticRegression().fit(z_cal.reshape(-1, 1), y_cal)
a, b = float(platt.coef_[0][0]), float(platt.intercept_[0])
print(f"slope a = {a:.3f}, intercept b = {b:.3f}")
z_test = np.log(np.clip(p_test, eps, 1 - eps) / (1 - np.clip(p_test, eps, 1 - eps)))
p_platt = platt.predict_proba(z_test.reshape(-1, 1))[:, 1]
print("platt ECE:", ece(y_test, p_platt))
# 2b. isotonic
iso = IsotonicRegression(out_of_bounds="clip").fit(p_cal, y_cal)
p_iso = iso.predict(p_test)
print("isotonic ECE:", ece(y_test, p_iso))
# 3. the reliability table itself
obs, pred = calibration_curve(y_test, p_test, n_bins=10, strategy="uniform")
for pr, ob in zip(pred, obs):
print(f"predicted {pr:.3f} observed {ob:.3f} gap {pr - ob:+.3f}")Scikit-learn also provides CalibratedClassifierCV, which wraps a classifier and handles the cross-fitting for you. It is the convenient option; check the signature for how a pre-fitted estimator is passed in your installed version, because that argument has changed across releases and passing it wrongly silently refits the base model.
What calibration does not fix
- It does not improve ranking. Every method here is monotone, so AUC is unchanged to within numerical noise. If the model cannot separate the classes, calibrating it produces honest probabilities that are all close to the base rate, which is correct and useless.
- It does not survive distribution shift. The correction is fitted on one population. When the input distribution moves, the calibration goes with it — so calibration belongs in the monitoring set and gets refitted, and it is often the cheapest response to drift because it needs no retraining.
- It does not fix a broken base rate. If the training set was resampled to balance the classes, the model’s probabilities are on the resampled base rate rather than the real one. Calibrating on a correctly-sampled held-out set does fix this, which is one more reason not to resample at all — threshold tuning usually beats resampling.
- It is not confidence per prediction. ECE is an aggregate over buckets. A model can be perfectly calibrated overall and badly miscalibrated on one subgroup; check the reliability table separately for the segments you care about, because the aggregate will hide a segment that is systematically wrong.