Automated Feature Selection Methods for Tabular Models

The three families of feature selection differ in one respect — how much of the model they consult — and that single difference predicts which features each one will keep, which it will drop, and exactly how it will be wrong.

What selection is actually for

Removing features rarely improves a boosted tree’s accuracy by much, and that is not the reason to do it. The reasons that survive contact with production are these. Inference cost is dominated by feature computation far more often than by the model, and a feature requiring a join against a slow service costs milliseconds on every request forever. Every feature is a pipeline that can break, and a model with 400 inputs has 400 upstream failure modes. Features degrade differently under drift, so fewer of them is a smaller surface. And neural networks genuinely do lose accuracy to uninformative columns, a finding covered in why neural networks underperform on tabular data — which means selection matters much more in front of a network than in front of a tree.

Guyon and Elisseeff’s “An Introduction to Variable and Feature Selection”, in the Journal of Machine Learning Research in 2003, is still the reference framing for the three families, and it makes the point that is easiest to forget: a variable that is useless by itself can be useful with others, and two redundant variables can be better than either alone. That is why the families disagree.

The worked feature set

Eight features predicting customer churn, with their Pearson correlation to the target and to each other:

                    corr with target
f1  tenure_months        -0.42
f2  monthly_charges       0.31
f3  total_charges        -0.20
f4  support_tickets       0.28
f5  tickets_last_30d      0.35
f6  is_month_to_month     0.40
f7  account_id            0.01
f8  seats_x_charges       0.29

Pairwise correlations that matter:
  f1 : f3   =  0.91     (tenure and lifetime spend move together)
  f4 : f5   =  0.74     (all-time and recent ticket counts)
  f2 : f8   =  0.88     (f8 is a product involving f2)
  f6 : f1   = -0.55
  everything else below 0.3

The set is deliberately realistic: one useless column (f7, an identifier that survived the schema), two strongly collinear pairs, and one engineered feature that is largely a restatement of an input.

Filter methods

A filter scores each feature against the target using a statistic that does not involve the model at all — correlation, mutual information, a chi-squared test, ANOVA F — ranks them, and keeps the top k. It is by far the cheapest option: one pass over the data, no training, and the cost is linear in the number of features.

Applied to the matrix above with a threshold of |corr| > 0.25, a filter keeps f1, f2, f4, f5, f6, f8 and drops f3 and f7. Dropping f7 is right. Dropping f3 is arbitrary — it correlates at 0.91 with f1, so it was always going to be redundant, but the filter dropped it for having a weaker marginal correlation rather than for being redundant, which means the decision was luck.

The structural weakness is that filters score features independently and so cannot see redundancy at all. They keep both f4 and f5, and both f2 and f8, because each scores well on its own. Two further limits: a linear correlation is blind to a non-linear relationship, so a feature with a perfect U-shaped relationship to the target scores near zero — use mutual information rather than correlation if you suspect this. And a univariate filter cannot find a feature that only matters in interaction, which is Guyon and Elisseeff’s point above and the reason a filter should be used to remove obvious junk rather than to choose a final set. Redundancy is its own topic, handled in redundant feature detection.

Wrapper methods

A wrapper treats the model as a black box and searches over subsets, scoring each by cross-validated performance. Recursive feature elimination is the common form: fit on all features, drop the least important, refit, repeat.

RFE on the eight features, 5-fold CV at each step:

  8 features   CV AUC 0.842
  drop f7  →   CV AUC 0.843     (identifier removed, tiny gain)
  drop f3  →   CV AUC 0.843     (redundant with f1, no loss)
  drop f8  →   CV AUC 0.841     (redundant with f2, slight loss)
  drop f4  →   CV AUC 0.840     (redundant with f5)
  drop f2  →   CV AUC 0.831     (real loss — stop here)

  selected: f1, f5, f6, f2  →  4 features, AUC within 0.002 of all 8

Wrappers find redundancy correctly, because a feature’s value is assessed in the presence of the others. That is the whole advantage. The costs are severe: an exhaustive search over n features is 2ⁿ subsets, RFE is n model fits times the CV folds, and each fit is a full training run. On 400 features with 5-fold CV, that is 2,000 fits. Worse, every one of those fits sees the validation data through the selection decision, so the reported score of the selected subset is optimistic unless the entire selection procedure is nested inside an outer cross-validation loop — which multiplies the cost again and is skipped almost universally.

Embedded methods

Embedded methods get selection as a side effect of fitting. L1 regularisation drives coefficients to exactly zero, so a lasso fit is simultaneously a model and a selected subset. Tree ensembles produce importances, and thresholding them selects. The cost is one training run.

On the worked set, an L1 model faced with the f1/f3 pair at 0.91 correlation typically zeroes one of them — and which one is unstable under resampling, so two runs on bootstrapped data can return different subsets from the same information. That instability is itself the finding: it tells you the pair is interchangeable, which is more useful than either answer.

Tree importances carry a specific bias that is worth knowing before trusting them. Carolin Strobl and co-authors showed in BMC Bioinformatics in 2007 that random forest variable importance measures are biased toward variables with many categories and toward continuous variables, since a high-cardinality column offers more candidate split points and can reduce impurity by chance. On the worked set this is exactly what would rescue f7: an identifier with 100,000 distinct values can pick up a non-trivial impurity-based importance while carrying no signal. Permutation importance — shuffle a column, measure the drop in held-out score — does not share this bias and is the safer default, at the cost of one extra evaluation pass per feature. Ranking features without training a model covers the cheap end of this spectrum.

Reading the disagreement

Run all three and the results differ. Filters keep f4 and f5; the wrapper keeps one; L1 keeps whichever it landed on. The instinct is to pick a winner. The better move is to read the pattern, because each kind of disagreement means something specific.

• All three keep it. It is signal. Keep it and stop thinking about it.
• All three drop it. It is noise. f7 should never have been in the table — it is an identifier, and the type detection that should have caught it is in automatic column type detection.
• The filter keeps it, the wrapper drops it. It is redundant, not useless. Which member of the redundant group you keep should be decided on operational grounds — keep the one that is cheaper to compute, more stable under drift, or easier to explain to a regulator.
• The filter drops it, the wrapper keeps it. It only matters in interaction. This is the most interesting case and the one a filter-only pipeline silently throws away.

A practical sequence that costs little: filter first to remove the obvious junk and get the feature count down to something a wrapper can afford, run an embedded method to get importances on what remains, and use a wrapper only on the top few dozen where the search is tractable. Then check the survivors for collinearity, which is a separate question with its own diagnostics in multicollinearity detection. And whatever the procedure, run it inside the training fold. A selection made by looking at the full dataset has used the validation labels to choose the features, and the score you report afterwards is not an estimate of anything.