Model Drift and When to Retrain

A model does not decay. The world moves and the model stays exactly where it was, which produces three distinguishable failures with three different monitors — and only one of the three can be detected before the labels arrive. That asymmetry, not the choice of statistical test, is what decides how long you spend wrong.

Three different failures

The distinction is not academic, because the remedies differ. Data drift may need nothing at all, or may need the model refitted on the new population. Concept drift needs a refit on recent data and probably a rethink of the features. Label drift often needs only recalibration, which is hours of work rather than weeks.

There is also a fourth thing that looks like drift and is not: a broken pipeline. An upstream schema change that turns a feature into nulls produces a dramatic drift signal and is fixed by an engineer, not by retraining. Check for it first, every time — it is more common than genuine drift and infinitely cheaper to fix.

The monitors you can run without labels

Order the monitors by what they require, because that determines how fast they can tell you anything.

1. Input distribution monitoring — available immediately. Per feature: null rate, cardinality, mean, key quantiles, and a distance from the training distribution (PSI or KS). Catches pipeline breaks and population shifts on the day they happen. Cannot see concept drift at all.
2. Prediction distribution monitoring — available immediately. The distribution of the model’s own output scores, and the rate at which they cross your decision threshold. Cheap, one series, and it catches the composite of every input change weighted by how much the model cares. If you monitor one thing, monitor this.
3. Proxy outcome monitoring — available in hours or days. Something correlated with the label that resolves fast: manual review agreement rate, first-week payment behaviour, click-through on a recommendation. Not the metric, but early.
4. Performance monitoring — available when labels are. AUC, precision at your operating threshold, calibration. The only monitor that sees concept drift, and the slowest one you have.

A drift test, worked

The two-sample Kolmogorov–Smirnov statistic is the maximum gap between two cumulative distributions. It needs no assumptions about shape and is one pass over sorted data.

Feature binned into five bands. Cumulative share in each:

band    train cum    live cum    |difference|
  1        0.20         0.32        0.12
  2        0.45         0.58        0.13   <-- maximum
  3        0.70         0.79        0.09
  4        0.88         0.93        0.05
  5        1.00         1.00        0.00

KS statistic = 0.13

Critical value at alpha = 0.05 for two samples:

  D_crit = 1.36 * sqrt(1/n1 + 1/n2)

  n1 = n2 = 5,000:
  D_crit = 1.36 * sqrt(2/5000) = 1.36 * 0.0200 = 0.0272

0.13 >> 0.0272, so the shift is significant.

Why statistical significance is the wrong alarm

Now run the same test at production volume. With 500,000 rows per side:

  D_crit = 1.36 * sqrt(2/500000) = 1.36 * 0.002 = 0.0027

A KS of 0.004 -- a shift far too small to change any decision --
is now "significant". At 5 million rows per side, D_crit = 0.00086.

Significance measures how confident you are that the distributions
differ at all. With enough data, they always differ, because nothing
in production is stationary. Confidence is not the question.

So alarm on effect size, not on p-values. PSI with its conventional bands (below 0.1 stable, 0.1 to 0.25 investigate, above 0.25 material) is one workable convention; a KS threshold set from what actually preceded past incidents is better, if you have the incidents to fit it on. The principle is the same one in anomaly detection: the alert threshold is a budget decision, not a statistical one.

The strongest version of this ties the alarm to a decision. Take last month’s traffic, replay it through the model, and count how many rows would cross the threshold differently under the shifted distribution. If the answer is eleven rows, there is nothing to do however significant the test.

Label delay sets your detection latency

This is the part that decides whether drift monitoring works, and it is almost never the part that gets discussed. Concept drift is only visible in performance, performance needs labels, and labels arrive late.

How many resolved cases does it take to see a real drop?

Precision falls from 0.60 to 0.50 -- a large, expensive change.
Detecting a difference of two proportions at 5% two-sided, 80% power:

  need  d  >  2.80 * sqrt( p1(1-p1)/n + p2(1-p2)/n )
        0.10 > 2.80 * sqrt( (0.24 + 0.25) / n )
        0.10 > 2.80 * 0.7 / sqrt(n)
        sqrt(n) > 19.6
        n > 385 per period

So roughly 385 resolved cases in the "before" window and 385 in the
"after" window.

Now add the delay. A lending model whose default label matures at
90 days, flagging 300 cases a month:

  drift begins                                    day   0
  affected cases accumulate (385 at 300/month)    day  39
  their labels mature                             day 129
  the difference reaches significance             day 129

Four months of bad decisions before the dashboard can possibly know.

Nothing about the statistics fixes this. What fixes it is finding a proxy that resolves sooner — first-payment behaviour, review-queue agreement, a small sample sent for immediate manual adjudication — and accepting that the proxy is worse in exchange for arriving in days. Budget the cost of a fast manual label stream against four months of undetected drift and it is usually not close.

Retraining policies, and their failure modes

• Scheduled. Refit weekly or monthly on a rolling window. Simple, predictable, and testable. The failure mode is that it is either too slow for a real shift or too fast for a stable problem — and a needless refit is not free, because it introduces a new model with new behaviour and no reason.
• Triggered by performance. Refit when the monitored metric drops past a threshold. Correct in principle and gated by the label delay above, which frequently makes it useless in practice.
• Triggered by drift. Refit when input or prediction drift exceeds a threshold. Fast, and wrong whenever the drift is benign — refitting on a population shift that did not change the relationship spends risk for nothing.
• Continuous / online. Update the model as data arrives. Necessary for a few domains, and the most dangerous option: a feedback loop where the model’s own decisions shape its training data will run away, and it does so quietly.

The policy that works for most teams is a scheduled refit with a drift-triggered early refit, and — this is the part that matters — a gate before deployment. Every candidate model, however it was triggered, is evaluated against the incumbent on a fixed recent holdout before it replaces anything. Automatic retraining with automatic deployment and no gate is how a broken upstream feed becomes a broken model at four in the morning.

A monitoring set that fits on one screen

1. Volume. Predictions served per hour, against last week. A dropped integration shows up here first and nowhere else.
2. Null and default rate per feature. One line each. Catches the schema change that turns a feature into a constant.
3. Prediction distribution. Mean score and the rate of crossing the decision threshold, daily. The single most informative series available without labels.
4. Input drift. PSI per feature against the training baseline, weekly. Sorted descending, top five shown.
5. Performance and calibration. On whatever labels have matured, with the maturation lag stated on the chart so nobody reads a stale window as current.
6. Decision impact. How many decisions changed relative to a replay under the training distribution. The number that converts every metric above into something a business owner can act on.