Handling Missing Values Before Training a Tabular Model

The default answer — fill with the mean — is defensible under one assumption about why the values are absent, and that assumption is usually false. Which method is correct is decided before any of them, by the missingness mechanism.

First ask why it is missing

Donald Rubin’s 1976 Biometrika paper “Inference and missing data” gave the three-way classification that every subsequent method rests on, and it is worth stating precisely because the names are routinely used loosely.

• MCAR — missing completely at random. Whether a value is missing is independent of everything, observed and unobserved. A sensor dropped packets at random. Under MCAR, dropping the incomplete rows is unbiased and only costs you sample size.
• MAR — missing at random. Missingness depends on observed data but not on the missing value itself. Income is missing more often for younger respondents, and age is recorded. This is the assumption under which model-based imputation is principled, because the observed columns carry the information needed to predict what is absent.
• MNAR — missing not at random. Missingness depends on the value that is missing. High earners decline to state their income. No imputation method can recover this from the data, because the information required is precisely what is not there.

The classification is not testable from the data alone — you cannot distinguish MAR from MNAR without knowing something about the collection process — which is why this is a conversation with whoever owns the pipeline rather than a statistical test. In practice the most common real mechanism in business data is MNAR and structural: the field is empty because it does not apply, and “no second address line” is not a missing value at all.

One column, three treatments

Take a column household_income from a 1,000-row survey with 200 values missing. The 800 observed values have this shape:

observed n = 800
mean     = 52,000
std      = 24,000
variance = 576,000,000
missing  = 200 rows (20%)

Suppose, from the survey’s own metadata, that missingness is concentrated among respondents under 25 — age is fully observed, so this is MAR with respect to age.

Mean imputation and the variance it eats

Fill all 200 with 52,000. The mean of the column is unchanged, which is the property people cite in its favour. The variance is not, and the arithmetic is worth doing because the size of the effect surprises people:

After filling 200 of 1,000 rows with the mean:

  sum of squared deviations, observed rows  = 799 * 576,000,000
                                            = 460,224,000,000
  sum of squared deviations, imputed rows   = 200 * 0  =  0

  new variance = 460,224,000,000 / 999 = 460,684,684
  new std      = 21,464

  variance reduced by 20.0%
  std reduced from 24,000 to 21,464  (-10.6%)

Twenty per cent of the variance is gone, and it went because 200 rows now sit exactly on the mean with zero deviation. That is not a rounding artefact; it is the defining behaviour of the method. Three consequences follow. The standard error of any estimate computed from this column is now too small, so confidence intervals are too narrow and significance tests over-reject. Correlations with other columns are attenuated toward zero, because 200 rows contribute a constant. And a spike of 200 identical values appears in the distribution, which a tree will happily split on — producing a model whose behaviour at exactly 52,000 differs from its behaviour at 51,999 for no reason related to the world.

Median imputation shares all of this and is more robust to skew, which for income — a strongly right-skewed quantity — makes it the better of the two constants. Neither is defensible when the missingness carries information.

Chained equations

Multivariate imputation by chained equations, described by Stef van Buuren and Karin Groothuis-Oudshoorn in the Journal of Statistical Software in 2011, treats each column with missing values as a regression target and cycles through them. The procedure is: fill everything with a crude initial guess; then for each column in turn, discard its imputed values, fit a model predicting it from all the other columns using the rows where it is observed, and re-impute. Repeat the whole cycle several times until the imputations stabilise.

On the worked column, this predicts each missing income from age and the other observed fields, so a 22-year-old and a 55-year-old receive different values. That is exactly the information mean imputation throws away, and it is why MICE is the right tool under MAR.

Two details are frequently got wrong. The first is that the proper version of MICE is multiple imputation: it generates several completed datasets by drawing from the posterior rather than taking the conditional mean, you fit the model on each, and you pool the results with Rubin’s rules so the extra uncertainty from having imputed appears in the standard errors. Taking a single conditional-mean imputation reintroduces the variance-shrinkage problem in a milder form. scikit-learn’s IterativeImputer exposes this as sample_posterior, which defaults to False.

The second is that it is still marked experimental, so the import is gated:

from sklearn.experimental import enable_iterative_imputer  # noqa: F401
from sklearn.impute import IterativeImputer

imp = IterativeImputer(
    max_iter=10,              # library default
    initial_strategy="mean",  # library default
    sample_posterior=False,   # library default; True for proper MI
    random_state=0,
)

The cost is real: MICE fits one model per column per iteration, so it is far more expensive than a constant fill, and on a wide table it can dominate the cost of training the actual model. It also assumes the regression models are adequate — imputing a strongly non-linear relationship with linear regressors produces confident nonsense.

The missingness indicator

The third treatment is the one that is under-used and often the strongest: add a binary column household_income_missingalongside whatever fill you chose. It costs one column and it is the only method that preserves the fact of absence.

This matters because absence is frequently the most predictive thing in the row. A loan applicant who left employer blank, a patient with no recorded follow-up, a user who never completed the profile — the missingness is a behaviour, and under MNAR it is the only trace of the information you cannot recover. Adding the indicator does not fix MNAR, but it lets the model use the pattern, which is strictly better than discarding it.

There is a version of this you get for free. Gradient-boosted tree implementations generally learn a default direction for missing values at each split rather than requiring imputation at all, which is equivalent to letting the tree use missingness as a feature at every node. If you are using XGBoost or LightGBM, imputing first can be strictly worse than passing the nulls through, because you have replaced a signal with a constant. Test both; the comparison is cheap and the result is dataset-specific.

Where imputation belongs in the pipeline

Fit the imputer on the training fold only, and apply it to validation and test. An imputer fitted on the whole dataset has computed a mean — or an entire regression model, in the MICE case — from rows that are supposed to be unseen, and the validation score is optimistic by an amount you cannot estimate. It is the same structural error as fitting an encoder before splitting, described in detecting target leakage, and it is easy to make because the imputation step usually happens during cleaning, long before anybody thinks about splits.

Three closing rules. Never impute the target — drop rows with a missing label, because inventing one is inventing supervision. Record the fill strategy and its parameters as part of the model artefact, since inference must apply the identical transform. And treat a column above roughly 50% missing as a candidate for deletion or for replacement by its indicator alone; at that rate you are modelling the imputer more than the data. The split strategy that all of this has to survive is covered in tabular cross-validation strategies.