Uplift Modelling: Who Would Have Bought Anyway

A churn model ranks customers by how likely they are to leave. A retention campaign needs customers ranked by how much the offer changes what they do. These are different orderings, and using the first where the second is needed is how an accurate model funds a campaign that loses money. The arithmetic is below, in full.

The question a risk model does not answer

Write the two quantities out and the problem is immediate.

risk model estimates:      P(churn | customer)

uplift model estimates:    P(stay | customer, treated)
                         - P(stay | customer, not treated)

The second is a difference between two worlds, only one of which happens to any given customer. That is the fundamental problem of causal inference and it is why uplift is harder than it looks: the target variable is never observed for anyone. You cannot compute it per customer, ever, and no amount of data changes that. It can only be estimated over groups, which is why every honest uplift programme is built on a randomised experiment.

Four kinds of customer

Crossing what the customer does when treated with what they do when left alone gives four types. The names are standard.

A churn model sorts by the probability of leaving, which mixes persuadables and lost causes together at the top and cannot separate them, because from the risk model’s point of view they look identical. They are identical — until you intervene.

The campaign that loses £191,000

A subscription business with 100,000 customers. The churn model is good: it ranks honestly and the top decile churns at many times the base rate. The retention team takes the top 10,000 and sends each a £30 credit. Annual margin per retained customer is £200.

The composition of that top decile, and how each part responds — the numbers are invented to make the arithmetic legible, but the shape is the point:

segment          count    retain if     retain if      uplift    incremental
                          treated       not treated              customers

lost causes      6,000      0.05          0.05          +0.00           0
persuadables     1,500      0.60          0.20          +0.40        +600
sure things      2,000      0.98          0.97          +0.01         +20
sleeping dogs      500      0.55          0.70          -0.15         -75
                                                                   -------
                                                                     +545

VALUE   545 incremental retained x £200 margin        =  £109,000
COST    10,000 credits x £30                          =  £300,000
                                                         ---------
NET                                                     -£191,000

Every number in the model was right. The ranking was honest, the probabilities were calibrated, the top decile really did contain the churn. And the campaign destroyed £191,000, because 60% of the budget went to people who left anyway, 20% went to people who were staying anyway, and 5% went to people the contact actively pushed out.

Note also what the campaign report will say. Of the 10,000 treated, roughly 3,585 were retained — retention in the top-risk decile far above what the model predicted for them. Presented without a control group that is a triumph, and it will be presented that way.

The break-even uplift

The threshold that decides who is worth contacting comes straight out of the two numbers above and needs no model at all.

Contact a customer only if:

    uplift x value_of_retention  >  cost_of_treatment

    uplift  >  cost / value
    uplift  >  £30 / £200
    uplift  >  0.15

Any customer whose probability of staying rises by less than 15
percentage points because of the offer is loss-making to contact,
however likely they are to churn.

Check against the table:
  lost causes      0.00  ->  no      (0.00 x 200 = £0    vs £30 cost)
  sure things      0.01  ->  no      (0.01 x 200 = £2    vs £30 cost)
  sleeping dogs   -0.15  ->  no      (-0.15 x 200 = -£30 vs £30 cost)
  persuadables     0.40  ->  yes     (0.40 x 200 = £80   vs £30 cost)

One line of arithmetic disqualifies 85% of the list the churn model produced. This is the same structure as choosing a classification threshold from a cost matrix: the model produces a number, the economics produce a cut-off, and the cut-off is never 0.5 and never the top decile by default.

Ranking by uplift instead

Now rank the whole base by estimated uplift rather than by risk, and contact only those above the break-even. Suppose that finds 2,000 customers with an average uplift of 0.30 — drawn from across the risk distribution, not only from the top decile.

VALUE   2,000 x 0.30 x £200                        =  £120,000
COST    2,000 x £30                                =   £60,000
                                                      ---------
NET                                                   +£60,000

Same product, same offer, same underlying customers.
The change is which score is sorted on.

  risk-ranked   -£191,000
  uplift-ranked  +£60,000
  difference     £251,000

The smaller list is also the better list, which is the part that is hardest to sell internally. Every incentive in a marketing organisation pushes towards contacting more people, and uplift targeting almost always says contact fewer.

How uplift is actually estimated

All approaches need the same input: a randomised experiment where a random subset was treated and a random subset was not. Without randomisation you are estimating a correlation between being treated and staying, and the people who were treated were selected for a reason, which is exactly the confounding problem.

1. Two-model (T-learner). Fit one model on the treated group and one on the control group, both predicting retention. Uplift is the difference of the two predictions. Simple, works with any library, and has a real weakness: both models are fitted to predict retention, so their errors do not cancel and the difference between two noisy predictions is noisier than either.
2. Single model with a treatment interaction (S-learner). One model over both groups with a treatment flag as a feature, scored twice with the flag on and off. Fewer parameters and more stable, but a tree-based model will often ignore the treatment flag entirely if its effect is small relative to the main effects — which is exactly the case where you needed it.
3. Class transformation. For a 50/50 randomised split, define a new binary target Z = 1 when (treated and responded) or (not treated and did not respond). A standard classifier fitted to Z has the property that 2·P(Z=1) − 1 estimates the uplift directly. One model, ordinary tooling, no differencing. The assumption of equal treatment probability matters; with unequal split sizes the transformation needs reweighting.
4. Uplift trees. Trees whose splitting criterion maximises the difference in treatment effect between the child nodes rather than the purity of the outcome. This is the only family that optimises the quantity you care about directly, and it is what the dedicated libraries implement.

Evaluating a model of something unobservable

You cannot compute the error of an uplift prediction on a single customer, because the counterfactual is missing. Evaluation is therefore always group-level, and the standard instrument is the Qini or uplift curve.

1. Score every customer in a held-out randomised sample with the uplift model, and sort descending.
2. Walk down the sorted list. At each depth k, take the customers in the top k of both the treated and the control arms.
3. Compute the incremental response: the response rate among the treated in the top k, minus the response rate among the control in the top k, multiplied by the number of treated.
4. Plot that cumulative incremental response against k. A model with signal rises steeply, peaks partway down the list, and then falls — and where it peaks is your campaign size.

The peak is the output worth having. It is not a metric; it is the answer to “how many people should we contact”, read directly off a curve, and it is usually far fewer than anyone expected. A flat curve is a real and useful result too: it says the treatment effect does not vary in a way your features can find, and the honest next step is to test a different treatment rather than a different model.

Keep a permanent randomised holdout after launch. Uplift decays — the persuadables get persuaded — and without a control arm the campaign has no way to know. This is the same discipline as running the experiment properly in the first place.