Tabular Data Is Still Where Most ML Money Is

A tabular problem is one where each row is an entity and each column is something known about it. Almost every model in production that makes a decision about a customer, a transaction, a shipment or a claim is one of these, and almost all of them are gradient-boosted trees. The reason is structural rather than fashionable, and it has three parts.

What a tabular problem actually looks like

Between ten and a few hundred columns. Between a few thousand and a few hundred million rows. A mixture of numeric columns with wildly different units, low-cardinality categoricals, a few high-cardinality ones such as product or postcode, several timestamps, and a lot of missing values that are missing for reasons that carry information.

The columns are not exchangeable. In an image, any pixel is much like any other pixel and a rotation of the input is still the same picture. In a table, column 7 is days_since_signup and column 8 is plan_tier, and rotating them into each other produces nonsense. That single asymmetry is behind most of what follows.

The signal is also usually weak and spread thin. A churn model that reaches an AUC of 0.78 is a good churn model; there is no equivalent of the near-perfect accuracy that a well-fed image classifier reaches, because the outcome genuinely depends on things not in the table.

Seven task types, and what each one predicts

Most confusion in a tabular project is a task-type confusion: the team builds a classifier when the question is a time-to-event question, or a risk model when the question is an incremental-effect question. Sorting the work this way first is worth more than any algorithm choice.

Each of those has a page in this cluster. Uplift and survival are the two most commonly skipped, and both are usually skipped by building a binary classifier that answers a slightly different question well enough to look successful.

Three structural reasons trees win

That gradient-boosted decision trees generally beat neural networks on tabular benchmarks is a published empirical result — Grinsztajn, Oyallon and Varoquaux made the case in a 2022 NeurIPS datasets-and-benchmarks paper, and the ranking argument is covered in its own page. What that page argues, this one explains. There are three properties of the algorithms themselves, none of which depends on any benchmark.

1. Axis-aligned splits match how tabular signal is shaped

A decision tree cuts on one column at a time: tenure_days < 90. Tabular relationships very often are exactly that shape — a threshold effect on a single meaningful quantity, an interaction between two named columns, a step at a policy boundary. A neural network can represent a step function, but it has to learn it out of smooth building blocks, from limited data, whereas a tree gets it in one split.

The same asymmetry runs the other way. A boundary that is genuinely diagonal in two columns costs a tree a staircase of many splits. That is a real weakness and it is why an explicit ratio or difference feature so often improves a boosted model — you are handing it the rotation it cannot find. Feature engineering keeps its leverage on tables for this exact reason.

2. Scale and monotone transforms do not matter

A split at income < 42000 and a split at log(income) < 10.645 partition the rows identically. Trees are invariant to any monotone transform of any feature, which means no standardisation step, no sensitivity to a column measured in pennies next to a column measured in years, and near-total immunity to heavy-tailed distributions and outliers in the features. Gradient-based models are sensitive to all of it, and every one of those sensitivities is a preprocessing step that can be got wrong — often in a way that leaks the test set.

3. There is nothing to pretrain on

This is the reason most often missed, and it is the biggest. What made deep learning dominant elsewhere is transfer: a model trained on a billion images or a trillion tokens has learned representations that transfer to your task, so you start far from scratch. There is no equivalent for your table. Your plan_tier column does not mean what another company’s plan_tier means; the schemas do not align, the units do not align, and the semantics live in a data dictionary rather than in the numbers.

So a tabular model is trained from scratch on your rows only, which puts you back in the small-data regime where the model with the right inductive bias wins. Transfer learning is the idea that made everything else possible, and it is precisely the idea that does not apply here.

When trees are the wrong choice

• Free text or images in a column. A tree cannot read a support ticket. Turn it into features first — embeddings, an extracted label, a sentiment score — and feed those. That hybrid is the subject of putting a classical model and a language model in one system.
• Very high-cardinality interactions that must generalise. Recommendation over millions of items is a factorisation or embedding problem, not a splitting problem. A tree cannot invent a representation for an item it never saw.
• Extrapolation. A tree’s prediction is a leaf mean. Feed it a row past the edge of the training range and it returns the value from the edge, flat forever. For anything where the trend continues beyond the observed range — a price model on an inflating series, a growth forecast — a linear component is not optional.
• Smooth physical relationships with known form. If you know the law, fit the law. A regression with the right functional form beats a thousand trees approximating it with steps, and it extrapolates.
• Fewer than a few hundred rows. Boosting has enough capacity to memorise a small table before it learns anything. What to do with 400 rows is a different discipline.

The stack that most of this runs on

The working set is small and has been stable for years: pandas or Polars for the frame, scikit-learn for splitting, pipelines and metrics, and one of XGBoost, LightGBM or CatBoost for the model. Scikit-learn also ships its own histogram-based booster, HistGradientBoostingClassifier, which needs no extra dependency and handles categoricals and missing values natively.

from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_auc_score

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0
)

model = HistGradientBoostingClassifier(
    learning_rate=0.05,
    max_iter=500,
    early_stopping=True,
    validation_fraction=0.15,
    random_state=0,
)
model.fit(X_tr, y_tr)

print(roc_auc_score(y_te, model.predict_proba(X_te)[:, 1]))

That is a competitive tabular baseline in fifteen lines, and the honest position on most projects is that the remaining gain from model choice is a couple of points of AUC while the remaining gain from fixing the label definition is often twenty.

The hard part is never the model

Across the tasks above, the recurring failures are the same five, and none is an algorithm choice. The label is defined in a way that admits information from after the decision point. A column that only exists because the outcome already happened is in the feature set. The split is random when the data is temporal. The score is used at 0.5 because that is the default. And nobody checked whether the campaign built on the model made more money than not running it.

The rest of this cluster is those five, one at a time — starting with the six leakage patterns, which cause more silently useless models than every other cause combined.