Churn Prediction End to End

Almost every failed churn project failed at the label, not the model. “Which customers will churn” is not a question until you have said what churn is, when you are asking, and what happens between the asking and the answer. Get those three right and a default gradient-boosting model is enough; get them wrong and no model helps.

The label is the project

There are two kinds of churn and they need different definitions.

Contractual churn has an event: a subscription is cancelled, a contract is not renewed, a card is removed. The date exists in a table and the label is a lookup. The difficulty here is entirely about timing, which is the next section.

Non-contractual churn has no event. A retail customer does not resign; they simply stop coming. There is nothing in the database to join to, so churn has to be defined — and every definition is arbitrary. “No purchase in 90 days” is a choice, and a different choice produces a different label, a different base rate and a different model.

Pick the threshold from the observed repurchase distribution rather than from a round number: find the inactivity gap beyond which historical customers rarely returned. If 92% of customers who went 60 days without purchasing never purchased again, 60 days is defensible. If the distribution has no such elbow — which is common for genuinely irregular purchases — then binary churn is the wrong framing entirely and time to event is the right one.

Three windows, and the gap between them

Every training row is one customer at one moment. That moment splits time into three intervals, and every one of them has to be stated explicitly.

     <---- observation ---->|<- gap ->|<---- label window ---->
     ...........................|.........|........................
     features computed          |         |  did the customer churn
     ONLY from this period      |         |  during this period?
                                ^
                          prediction date

Example, monthly scoring:
  observation window   90 days ending on the prediction date
  gap (blackout)        7 days
  label window         90 days, starting 8 days after the prediction date

The gap is the part that is almost always missing, and it is the difference between a model that works and one that is useless in production.

A model with no gap learns to predict churn that is already happening. Its best features become things like “logged in zero times this week” — which is true, predictive, and arrives far too late for anyone to act on. By the time the retention team calls, the customer left a fortnight ago. Inserting a blackout of at least the length of your intervention cycle forces the model to find signal that exists early enough to be useful. It will score worse. It will also be worth something.

The label window must have a fixed length, and rows whose label window extends past the end of your data must be dropped. Keeping them means labelling a customer “retained” because their churn has not had time to happen yet, which teaches the model that recent customers do not churn.

Five traps inside the label

• Survivorship in the customer list. Building the training set from SELECT * FROM customers WHERE status = 'active' as of today removes everyone who already churned. The base rate collapses, the model looks excellent on a population that cannot contain the outcome, and it has never seen a churner from more than a year ago. Build the cohort as of the prediction date, from history.
• Cancellation-adjacent features. support_tickets_about_billing, visited_pricing_page, downgrade_requested. These are not causes of churn; several of them are steps in the cancellation flow. A model built on them scores beautifully and is describing the exit interview.
• A status column that was updated in place. If customers.plan_tier holds the current value rather than a history, then a customer who downgraded last month has their post-churn value in a feature that is supposed to describe the observation window. This requires either an event log or slowly-changing-dimension tables; there is no way to reconstruct it from a mutable row.
• A random split of a customer panel. If each customer contributes twelve monthly rows, a random split puts the same customer in train and test. The model recognises the customer rather than the behaviour. Split by customer with GroupShuffleSplit or GroupKFold, and split by time as well.
• Aggregates computed over the whole table. df["segment_churn_rate"] = df.groupby("segment")["churned"].transform("mean") puts each row’s own outcome into its own feature. Out-of-fold encoding or nothing.

All five are instances of the same failure, and the leakage page covers the general form with the code for each.

Features that exist at prediction time

The test for every feature is one question: could this value have been computed, from data that existed, at 00:00 on the prediction date? If the answer needs a caveat, drop it.

The pipeline

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.model_selection import GroupKFold
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OrdinalEncoder

# df has: customer_id, prediction_date, <features>, churned (0/1)
CUT = pd.Timestamp("2026-01-01")
train = df[df.prediction_date < CUT]
test = df[df.prediction_date >= CUT]      # time split, never random

cat = [c for c in features if df[c].dtype == "object"]
num = [c for c in features if c not in cat]

pipe = Pipeline([
    ("prep", ColumnTransformer([
        ("cat", OrdinalEncoder(handle_unknown="use_encoded_value",
                               unknown_value=-1), cat),
        ("num", "passthrough", num),
    ])),
    ("clf", HistGradientBoostingClassifier(
        learning_rate=0.05, max_iter=600, early_stopping=True,
        validation_fraction=0.15, categorical_features=list(range(len(cat))),
        random_state=0,
    )),
])

# cross-validate WITHIN the training period, grouped by customer
gkf = GroupKFold(n_splits=5)
scores = []
for tr, va in gkf.split(train[features], train.churned, groups=train.customer_id):
    pipe.fit(train[features].iloc[tr], train.churned.iloc[tr])
    p = pipe.predict_proba(train[features].iloc[va])[:, 1]
    scores.append(average_precision_score(train.churned.iloc[va], p))
print("cv average precision:", np.mean(scores), "+/-", np.std(scores))

pipe.fit(train[features], train.churned)
p_test = pipe.predict_proba(test[features])[:, 1]
print("holdout AP:", average_precision_score(test.churned, p_test))
print("holdout AUC:", roc_auc_score(test.churned, p_test))

Two things in that code do the real work. GroupKFold on customer_id stops the same customer appearing on both sides of a fold. The date cut on prediction_date stops the model being evaluated on a period it was trained through. Remove either and the reported score goes up and the deployed model gets worse.

Evaluating a rare-event model

Churn base rates of 2–5% per period are normal, and at that imbalance AUC-ROC is misleading in a specific way: it is dominated by the vast majority of easy negatives and stays high even when the top of the ranking is poor. Average precision — the area under the precision-recall curve — is the metric that moves when the top of the list changes, and the ROC and PR page explains exactly when imbalance decides which to use.

The number to put in front of the business is lift in the top decile. If the base rate is 3% and the top 10% of scored customers churn at 15%, the model concentrates five times the churn into a tenth of the population — a statement a retention manager can act on, unlike an AUC.

Then check calibration before anyone multiplies a score by a revenue figure. A boosted model’s 0.8 is very often not 80%, and expected value computed on uncalibrated scores is arithmetic on the wrong numbers.

The model is accurate and the campaign loses money

This is where most churn projects end, and it is the failure that the model cannot see. Suppose the model is good: the top decile churns at five times the base rate and the ranking is honest. The retention team sends everyone in it a discount. The campaign loses money anyway.

It loses money because the highest-risk customers are largely the ones who are leaving whatever you do — a discount does not fix a company that stopped needing the product — while the customers a discount actually saves sit in the middle of the risk distribution. Ranking by probability of churn is not ranking by responsiveness to intervention, and the uplift page works the arithmetic of a campaign losing £191,000 behind an entirely accurate churn model.

Build the churn model. It is the right first artefact, it tells you where the risk is, and it is the input to the segmentation. Then, before spending anything on the list it produces, run a randomised holdout on a slice of the target population so you can measure the incremental effect rather than assume it. That holdout costs a few thousand pounds and is the only thing that will ever tell you whether the project worked.