Gradient Boosting: Why XGBoost Still Wins on Tables

Gradient boosting is the least glamorous algorithm still winning real work, and the reason it wins on tables is not that trees are magic. It is that the inductive bias of an axis-aligned split matches what tabular data looks like, and neural networks have the wrong one.

One boosting round, by hand

The recipe: predict something simple, look at what is left over, fit a small model to the leftovers, add a shrunken version of it, repeat.

Four rows, targets 10, 12, 30, 32, with one feature that happens to separate the first two from the last two.

F₀ = mean(y) = 21                      predictions: 21, 21, 21, 21
residuals r = y − F₀                   →  −11, −9, +9, +11

fit a depth-1 tree (a "stump") to r
  left leaf  (rows 1,2) = mean(−11, −9) = −10
  right leaf (rows 3,4) = mean(+9, +11) = +10

F₁ = F₀ + ν · stump,  with learning rate ν = 0.1
  rows 1,2:  21 + 0.1 × (−10) = 20.0
  rows 3,4:  21 + 0.1 × (+10) = 22.0

new residuals: −10, −8, +8, +10          (all shrank)

Notice the shrinkage. Without ν the first stump would jump straight to 11 and 31 and the ensemble would be one crude tree. With ν = 0.1 each tree contributes a tenth, so it takes hundreds of rounds and each round can correct the last — the same bias-variance bargain as everywhere else, with ν and the number of rounds trading off almost exactly against each other. Halve the learning rate, roughly double the trees.

Where the gradient comes in

The residual y − F is not an arbitrary choice. For squared error, L = ½(y − F)², so ∂L/∂F = −(y − F). The residual is the negative gradient of the loss with respect to the current prediction. Boosting is gradient descent in function space: each tree is one step, and ν is the step size.

That generalisation is Friedman’s (Annals of Statistics, 2001, “Greedy function approximation: a gradient boosting machine”), and it is what makes the method work for any differentiable loss. For log-loss the tree is fitted to y − p — which, by the cross-entropy derivation, is the same “prediction minus target” quantity that appears at the output layer of a neural classifier. Two very different model families, one gradient.

What the published comparisons report

The claim “trees beat deep learning on tabular data” is worth handling carefully, because it is a claim about a body of evidence rather than a theorem. Two papers are the ones to cite.

• Grinsztajn, Oyallon and Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data?” (NeurIPS 2022, Datasets and Benchmarks track). They assemble a benchmark of medium-sized tabular datasets with a standardised hyperparameter search budget, report tree ensembles ahead of neural baselines, and — more usefully — run ablations designed to isolate why.
• Shwartz-Ziv and Armon, “Tabular data: Deep learning is not all you need” (Information Fusion, 2022). They re-evaluate several proposed deep tabular architectures on datasets other than the ones each paper introduced them with, and report that the advantage largely does not survive the change of dataset — a result about evaluation practice as much as about architectures.

Both are worth reading rather than quoting, and neither says neural networks cannot work on tables. They say that on medium-sized, heterogeneous tabular problems with a fixed tuning budget, the tree ensemble is the stronger default.

Three structural reasons

The ablations in the first paper point at properties of the model classes, and these are the parts that generalise beyond any benchmark:

• Neural networks are biased toward smooth functions. Tabular targets frequently are not smooth — a threshold at exactly €10,000 because that is where a reporting rule kicks in. A tree represents that with one split; a network has to approximate a step with a sum of smooth pieces.
• Uninformative features hurt networks more. A tree simply never splits on a useless column. A dense layer gives it a weight, which is one more thing to fit and one more source of variance.
• Trees are invariant to monotone transforms; networks are not. Log-scaling a feature does not change any split a tree would make, because splits depend only on order. The same transform changes a network’s optimisation substantially, which is why tabular networks need careful normalisation and trees need none.

There is an operational reason too, and it is not in any paper: a gradient-boosted model trains in seconds on a laptop, has three hyperparameters that matter, tolerates missing values natively, and produces feature importances you can argue with. On a two-week project, that gap is worth more than a point of accuracy.

Three hyperparameters carry most of the tuning value, and they are not independent. The learning rate ν and the number of trees trade off almost exactly against each other, so the usual approach is to fix ν low — 0.05 or 0.03 — and let early stopping on a validation fold decide the tree count rather than guessing it. Tree depth (or maximum leaves) controls how much interaction between features a single tree can express; six is a reasonable starting depth and anything past about ten is usually memorising. Subsampling rows and columns per tree adds randomness that behaves like a regulariser. Everything else in the parameter list is a refinement on those.

When a network does win

• The features are perceptual. Images, audio, raw text — the tree has nothing to split on that means anything.
• The dataset is very large and homogeneous. Tree ensembles stop improving before networks do when rows run into the tens of millions with consistent structure.
• You need one model over mixed modalities. If text, images and columns have to be reasoned about jointly, a network is the natural home — though the pragmatic version is usually to embed each modality separately and hand the vectors to the tree.
• The output is a sequence or a structure. Trees predict a number or a class, and that is all.

Practical note on the three implementations: XGBoost grows trees level-wise and is the most conservative default; LightGBM grows leaf-wise, which is faster and easier to overfit on small data; CatBoost handles high-cardinality categoricals with an ordered target statistic that avoids the leakage described in target encoding. All three will be within noise of each other on most problems once tuned, so pick on ergonomics.