Target Leakage: Finding a Feature That Already Knows the Answer

A model with an implausibly good validation score usually has a column that could not have existed at prediction time. The hard part is not fixing it — you drop the column — but telling it apart from a column that is simply very informative.

The definition that actually discriminates

Target leakage is the presence, in the training features, of information that would not be available at the moment the model is asked to predict. The test is temporal and operational, not statistical. A column leaks if its value is determined by, or written after, the event you are predicting.

This matters because the usual heuristic — “drop features that correlate suspiciously highly with the target” — is wrong in both directions. Some legitimate features are enormously predictive: a credit score genuinely predicts default, and dropping it for being too good throws away the model. And some leaks are weak individually: a last_updated_at timestamp that shifts by a day for closed cases leaks the outcome through a feature whose marginal correlation with the target is unremarkable, because the leak lives in an interaction that a boosted tree will happily find.

Kaufman, Rosset and Perlich set out the general formulation in their KDD 2011 paper “Leakage in Data Mining: Formulation, Detection, and Avoidance”, and their framing is the useful one: leakage is a mismatch between the data used to train and the data legitimately available at prediction time. Everything below is a way of finding that mismatch.

What it does to the model

Three distinct harms, and they are worth separating because only the first is obvious.

• The validation score becomes meaningless. Not slightly optimistic — meaningless, because the split does not remove the leaked information. Cross-validation cannot detect this; every fold contains the same leak.
• The model stops learning the real signal. Boosting is greedy. Given a column that resolves most of the loss in the first few splits, subsequent trees fit the residual noise rather than the genuine predictors. Remove the leak later and you do not get the model you would have trained without it; you have to retrain from scratch, and the honest model is often structurally different.
• Production degrades silently and unevenly. The leaked column still exists at serving time, it just carries a null, or a default, or a stale value. Trees route nulls to a learned default direction, so the model does not error — it predicts confidently and wrongly, and only on the rows where the field has not been populated yet, which are usually the recent ones.

The four places it gets in

Post-outcome fields on the same row. The classic: refund_amount when predicting fraud, cancellation_reason when predicting churn, discharge_date when predicting length of stay. These are obvious once named, and are usually found by reading the column list rather than by any statistic.

Aggregates computed over the whole history. A feature like customer_lifetime_orders, computed today and joined onto a row from eighteen months ago, encodes everything the customer did after that row. Any aggregate must be computed as of the row’s own timestamp, which is the entire reason point-in-time feature stores exist.

Preprocessing fitted before the split. Target encoding, imputation with a target-conditional statistic, feature selection scored on all rows, or resampling applied before splitting. This is leakage from the labels into the features, and the cure is structural — every fitted step inside the fold, as covered in cross-validation strategies for tabular models.

Aggregates computed over the whole cohort. A related but distinct case: a feature like “this customer’s spend relative to the population mean”, where the population mean was computed over all rows including the ones you are predicting. Each row now carries a trace of every other row’s target-correlated behaviour. The effect is small per row and it does not vanish under any splitter, because the leak is baked into the feature before splitting happens.

Identifiers and their proxies. A row id that is allocated in outcome order, a file path that sorts by class, an auto-incrementing key that correlates with a policy change. These leak through ordering rather than through content, and they survive being “anonymised” because the order is the leak.

The check: correlation plus timestamp

Correlation alone cannot separate leakage from strength. Correlation paired with a question about when the value was written can. Run the statistical screen to produce a shortlist, then adjudicate each candidate with the operational question.

1. Rank every column by association with the target — point biserial correlation for numeric, mutual information for mixed. See ranking features without training a model for the mechanics.
2. Take the top few and fit a single-feature model on each. Any column that reaches an AUC above roughly 0.95 alone is a candidate, not a triumph.
3. For each candidate, answer one question in writing: at the instant the model runs in production, is this field already populated with this value? If the answer needs a caveat, it leaks.
4. Where a timestamp exists for the field, compare it to the target event timestamp across the training set. Any row where the field was written after the event is proof.
5. Refit without the candidate. A score that collapses from 0.98 to 0.76 is not a loss — 0.76 is what the model was always worth.

A worked example that catches one

Suppose a churn table with monthly_spend, support_tickets_90d, tenure_months and account_status_updated_at. Single-feature AUCs come out at roughly 0.61, 0.66, 0.64 and 0.97. The fourth is a date field, which is why nobody looked at it — and it is the leak: the status timestamp is rewritten by the billing system on the day an account closes, so its recency is a near-perfect indicator of churn that did not exist when the prediction would have been needed.

import numpy as np, pandas as pd
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import cross_val_predict, StratifiedKFold
from sklearn.ensemble import HistGradientBoostingClassifier

cv = StratifiedKFold(5, shuffle=True, random_state=0)

# Step 1: what does each column achieve alone?
solo = {}
for col in df.columns.drop("churned"):
    x = pd.to_numeric(df[col], errors="coerce").to_numpy().reshape(-1, 1)
    p = cross_val_predict(
        HistGradientBoostingClassifier(max_iter=100, random_state=0),
        x, df["churned"], cv=cv, method="predict_proba",
    )[:, 1]
    solo[col] = roc_auc_score(df["churned"], p)

print(pd.Series(solo).sort_values(ascending=False).round(3))

# Step 2: for the suspect, is the field written after the event?
after = (df["account_status_updated_at"] > df["churn_event_at"]).mean()
print("share of rows where the field was written after the outcome:", round(after, 3))

The second number is the one that settles it. If a meaningful share of rows have the field written after the outcome, no amount of feature engineering rescues the column; it has to go, along with anything derived from it — a “days since last status change” feature carries exactly the same leak in a form that no longer looks like a date.

One caution about the shortlist step. A column can be redundant with a leaked column without being obviously temporal, so after dropping a leak, re-run the screen rather than assuming the job is done; near duplicates of the dropped column will move to the top of the ranking, which is what duplicate and redundant column detection is for.