Cross-Validation Strategies for Tabular Models

A cross-validation score is a prediction about a future you have not seen. It is only as good as the resemblance between how you split the data and how the model will actually be asked to generalise — and on most real tables, a shuffled k-fold does not resemble that at all.

What a split is actually simulating

Every validation scheme answers one question: when this model meets a row it has never seen, how wrong will it be? The word doing the work is unseen. A shuffled k-fold defines unseen as “a row that was not in the training partition”. Production defines it differently, and almost always more strictly: a customer who has never transacted before, a hospital not in the study, a week that has not happened yet, a sensor that was installed last month.

The gap between those two definitions is where inflated scores come from. If your validation rows differ from training rows only by index, while production rows differ by entity or by time, the score is measuring an easier problem than the one you have. Nothing about the model is wrong. The measurement is.

How grouped rows leak through k-fold

Grouped data means several rows share a source: multiple visits per patient, multiple orders per customer, multiple photographs of one part, multiple readings from one device. Those rows are not independent. They share the group’s idiosyncrasies, and often they share the label too.

Shuffle and split five ways, and a customer with twelve orders has roughly a one in five chance of landing entirely inside a single fold and an overwhelming chance of appearing in both the training and the validation partition. The model does not need to learn the general relationship between features and label. It can learn this customer usually churns, from the eleven of their rows it trained on, and score the twelfth correctly. That is memorisation rewarded as generalisation.

The effect is strongest exactly where you would least want it: high cardinality identifiers, models with enough capacity to memorise, and groups whose label is constant. A gradient boosted tree with a few hundred trees will find a leaf that isolates a group without being told the group id exists, because some combination of a postcode, a device model and a signup month is effectively an identifier.

A worked inflation

Take a table of 10,000 rows drawn from 1,000 groups, ten rows each, and suppose the label is constant within a group. Split it 5-fold, shuffled. Each fold holds 2,000 validation rows, and the training partition holds the other 8,000 — which contains, on average, 8 of the 10 rows of every group represented in validation. Effectively every validation row has eight labelled neighbours inside the training set that came from the same source.

Now split the same table with GroupKFold(n_splits=5). Each fold holds out 200 whole groups; not one of their rows appears in training. The two numbers you get are measuring different tasks. The first answers “can the model recognise a group it has already seen?” The second answers “can it handle a new customer?” If your product onboards new customers, only the second number has ever been relevant.

import numpy as np
from sklearn.model_selection import GroupKFold, KFold, cross_val_score
from sklearn.ensemble import HistGradientBoostingClassifier

rng = np.random.default_rng(0)
n_groups, per_group = 1000, 10
groups = np.repeat(np.arange(n_groups), per_group)

# A per-group quirk that is visible in the features and decides the label.
quirk = rng.normal(size=n_groups)
y = (quirk > 0).astype(int)[groups]
X = np.column_stack([
    quirk[groups] + rng.normal(scale=0.4, size=n_groups * per_group),  # noisy view of the quirk
    rng.normal(size=n_groups * per_group),                             # pure noise
])

model = HistGradientBoostingClassifier(random_state=0)
naive = cross_val_score(model, X, y, cv=KFold(5, shuffle=True, random_state=0))
honest = cross_val_score(model, X, y, cv=GroupKFold(5), groups=groups)
print("shuffled k-fold:", naive.mean().round(3))
print("group k-fold:   ", honest.mean().round(3))

Run it and the shuffled number is the higher one. The size of the gap depends on how much of the label the group quirk explains and how noisy the feature view of it is — which is why the gap is worth measuring on your own data rather than quoting from anyone else. The direction, though, is not in doubt: a shuffled split on grouped rows can only ever be optimistic, never pessimistic.

Matching the splitter to the data shape

• Independent rows, balanced classes: KFold(shuffle=True). This is the only case the default is right for.
• Imbalanced classes: StratifiedKFold, which preserves the class ratio in every fold. Without it, a 2% positive rate and five folds gives you folds whose positive counts vary enough to make the fold-to-fold variance in the metric larger than the effect you are trying to measure.
• Grouped rows: GroupKFold, or StratifiedGroupKFold when the classes are also imbalanced. The group key is whatever entity production will hand you fresh.
• Ordered rows: TimeSeriesSplit, which only ever trains on rows earlier than the validation window. Shuffling time-ordered data trains on the future, and no amount of feature hygiene fixes that.
• Both grouped and ordered: split by time first, then check that no group straddles the boundary. If groups persist across the cut, you have both problems at once and the time split alone will not save you.

scikit-learn documents the full set and, usefully, the visual comparison of how each one carves a dataset — the scikit-learn user guide on cross-validation is the primary reference and worth reading once end to end rather than reaching for the default.

Preprocessing belongs inside the fold

The second most common way a tabular score inflates has nothing to do with the splitter. It is fitting a transformer on all the data before splitting. A StandardScaler fitted on the full table has seen the validation rows’ mean; a SimpleImputer fitted on the full table fills training gaps with a statistic computed partly from validation; a target encoder fitted on the full table has poured the validation labels into a feature column, which is the most severe form of it.

There is a subtler variant that survives a correct pipeline: choosing the number of folds, the splitter, or the model family by looking at the cross-validation score, and then reporting that same score. Each such choice consumes a little of the validation set’s independence, and after fifty tuning runs the best CV score is partly a measurement of which configuration got lucky on these particular folds. A held-out set the search never touches is the only defence, and it has to stay untouched — looking at it twice makes it a validation set.

The fix is mechanical rather than clever: put every fitted step inside a Pipeline and pass the pipeline to cross_val_score, so each step is re-fitted on the training partition of every fold. Feature selection counts as a fitted step. So does outlier removal. So does anything that computed a number from the labels.

Once the splitter matches the data and the preprocessing is inside the fold, the remaining sources of a too-good score are in the columns themselves — see finding a feature that already knows the answer. And if the score is unstable rather than inflated, the label column is worth a look before the model is: disagreement between folds is itself a mislabelling signal.