Train, Validation and Test: Why Three Splits
5 min read · updated August 3, 2026
Everyone knows to hold out a test set. Far fewer can say why a third split is needed, and the reason is not tradition — it is that looking at a number and acting on it is a form of fitting, and it costs you exactly as much as it sounds like it should.
What each split is actually for
- Train. Fits the parameters. The optimiser sees this and nothing else.
- Validation. Fits the decisions: which architecture, which features, which regularisation strength, which threshold, when to stop. Nobody calls this fitting, and it is.
- Test. Estimates what you will get in production. Valid exactly as long as it has influenced no decision. Look at it twice and it has become a second validation set.
The rule that follows is unpopular and correct: the test set is consumed by use. If you evaluate on it, adjust something and evaluate again, its number is now optimistic and there is no way to un-see it. Teams that ship well treat it like a sealed envelope, and hold a second one for the day someone insists.
Split ratios matter much less than people think, and the reason is the arithmetic in the next section: what counts is the absolute size of the held-out sets, not the percentage. At 200 rows a 70/15/15 split leaves thirty examples to decide on and thirty to report from, and both numbers are noise — cross-validation is the answer there, not a different ratio. At ten million rows, 1% is a hundred thousand examples and is ample for both. Work backwards from the smallest difference you need to be able to detect.
How much optimism selection buys
Here is the arithmetic nobody does, and it is short. A validation accuracy measured on n examples is a binomial proportion, so its standard error is √(p(1−p)/n). At p = 0.80 and n = 500:
SE = √(0.80 × 0.20 / 500) = √0.00032 = 0.0179 → 1.8 percentage points
Now suppose you try 20 configurations that are, unknown to you, genuinely identical in quality. You will report the best one. The expected maximum of 20 draws from a standard normal is about 1.87, so the winner scores roughly 1.87 × 1.8 ≈ 3.4 points above the truth — from noise alone, with no model being better than any other.
Assumptions, labelled: the configurations are equally good, the validation errors are independent across configurations (in practice they are correlated, which shrinks the effect), and the normal approximation to the binomial holds at this n. The conclusion survives all three being loosened. A 3-point “improvement” found by sweeping twenty settings on a 500-row validation set is indistinguishable from luck, and the test set is what tells you which it was.
The same arithmetic is why eval suites need a sample size before they need a metric, and why a leaderboard that moves by half a point between model releases is reporting weather.
Six ways a split leaks
Leakage is when information about the held-out answer reaches the model. It always shows up the same way — offline results that are too good, and production results that are not — so learn it by name.
| Leak | Description |
|---|---|
| target leakage | A feature that only exists because the outcome already happened: 'number of collection letters sent' when predicting default, 'discharge summary text' when predicting readmission. Symptom: one feature dominates and accuracy is suspiciously high. |
| temporal leakage | Random splitting of time-ordered data, so the model trains on the future and is tested on the past. Symptom: excellent offline results, immediate decay in production. Fix: split by date, always. |
| group leakage | The same entity — patient, customer, document, repository — appears in both splits, so the model recognises rather than generalises. Fix: split by group id, not by row. |
| preprocessing leakage | Fitting a scaler, imputer, encoder, feature selector or PCA on all the data before splitting. The held-out rows contributed to the transform. Subtle, extremely common, and usually worth a point or two of fake accuracy. |
| duplicate leakage | Near-duplicate rows straddling the split — scraped pages, re-posted records, augmented copies. Deduplicate before splitting, on a normalised form rather than exact equality. |
| selection leakage | The held-out set was filtered by something correlated with the label — only completed orders, only rows with no missing fields. The estimate is honest for a population you will never see. |
The fix is structural
You do not fix leakage by being careful, because being careful does not survive the fourth notebook. You fix it by making the split the first operation and putting every transform inside the fitted object, so that fitting on held-out data is not something the code can express:
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.model_selection import GroupKFold, cross_val_score
pipe = make_pipeline(SimpleImputer(), StandardScaler(), estimator)
# every fold refits the imputer and the scaler on that fold's training part only
scores = cross_val_score(
pipe, X, y,
cv=GroupKFold(n_splits=5),
groups=customer_id, # no customer spans two folds
)Two things are doing the work there. The pipeline means the imputer and scaler are fitted inside each fold, closing the preprocessing leak. GroupKFold with a customer id closes the group leak. For time series, replace it with a forward-chaining splitter so every evaluation is on data later than everything it trained on.
The same failure at internet scale
Benchmark contamination is duplicate leakage with a web crawl as the copying mechanism: the test items are on the public internet, the public internet is in the training corpus, and the resulting score measures recall of the answer key. Everything above applies, with the extra difficulty that you cannot inspect the training set — which is why contamination is detected statistically rather than by looking.
The practical version for anyone shipping an LLM feature: your eval set has to be data the model could not have seen, which in practice means data you wrote or data from after a cutoff, kept out of every prompt and every fine-tune. That is the same sealed envelope, with a harder envelope.