Stacking Gradient Boosting Models: When the Extra Complexity Pays For Itself

Stacking reliably improves a tabular score by a small amount. Whether that small amount is worth doubling your inference cost, your training time and the number of models you have to monitor is a question with a different answer for a Kaggle leaderboard and for a production service.

What stacking is, mechanically

Train several base models. Collect their predictions. Train a second model — the meta-learner, or blender — that takes those predictions as its input features and outputs the final answer. That is all of it. David Wolpert introduced the scheme as “stacked generalization” in Neural Networks in 1992, and Leo Breiman adapted it to regression in “Stacked Regressions”, published in Machine Learning in 1996 — where he also established the point that made it work in practice: the meta-learner should be constrained, and non-negativity constraints on the weights markedly improve it.

Distinguish it from the two things it is often confused with. Blending with fixed weights — averaging two models, or taking 0.7 of one and 0.3 of the other — is not stacking, because nothing is learned. Bagging trains the same algorithm on resampled data. Stacking learns how to combine different algorithms, and the learned part is what creates both its advantage and its main failure mode.

Out-of-fold predictions are the whole trick

If you train a base model on all the training rows and then feed its predictions on those same rows to the meta-learner, you have built a leak. The base model’s in-sample predictions are far better than its out-of-sample ones — a boosted tree can fit training rows very closely — so the meta-learner learns to trust a level of accuracy that will not exist at serving time. It over-weights the model that overfits most, which is precisely backwards.

The fix is out-of-fold prediction. Split the training data into k folds. For each fold, train the base model on the other k−1 and predict this one. Every training row now has a prediction from a model that never saw it, and those are the meta-learner’s features. Then refit each base model on all the data for use at serving time.

from sklearn.ensemble import StackingClassifier, HistGradientBoostingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from lightgbm import LGBMClassifier

cv_inner = StratifiedKFold(5, shuffle=True, random_state=0)
cv_outer = StratifiedKFold(5, shuffle=True, random_state=1)

stack = StackingClassifier(
    estimators=[
        ("hgb",  HistGradientBoostingClassifier(random_state=0)),
        ("lgbm", LGBMClassifier(num_leaves=63, n_estimators=600,
                                learning_rate=0.05, random_state=0, verbose=-1)),
    ],
    final_estimator=LogisticRegression(C=1.0, max_iter=1000),
    cv=cv_inner,              # this is what makes the meta-features out-of-fold
    stack_method="predict_proba",
    passthrough=False,        # try True to also give the meta-learner raw features
    n_jobs=-1,
)

base  = cross_val_score(HistGradientBoostingClassifier(random_state=0),
                        X, y, cv=cv_outer, scoring="roc_auc")
mixed = cross_val_score(stack, X, y, cv=cv_outer, scoring="roc_auc")
print("single:", base.mean().round(4), "+/-", base.std().round(4))
print("stack: ", mixed.mean().round(4), "+/-", mixed.std().round(4))

Two things about that comparison. The outer CV must be separate from the inner one, or you are measuring the stack on data used to build its meta-features — the same nested-validation requirement described in cross-validation strategies for tabular models. And compare the standard deviations, not just the means: a gain smaller than the fold-to-fold spread has not been demonstrated.

Where the gain comes from, and where it does not

Stacking helps to the extent that the base models make different errors. Two boosted-tree models with different seeds and slightly different depths make nearly the same errors on nearly the same rows; the combination has little to average away, and the meta-learner cannot manufacture information that neither model had.

Diversity that actually pays comes from different inductive biases: a boosted tree plus a regularised linear model, or plus a k-nearest neighbour model, or plus a neural network on the same table. The tree handles interactions and non-monotone effects; the linear model extrapolates outside the training range, which trees fundamentally cannot do, since a tree’s prediction is constant beyond its outermost split. On a table where extrapolation matters, that pairing is a genuine complement rather than two views of the same thing. The reasons a neural model is usually not the best single choice on tabular data, but can still contribute diversity, are covered in the gap between neural nets and trees on tabular data.

Set expectations from the mechanism rather than from a number somebody quoted. If a single tuned boosted tree is already near the Bayes error for the features you have, there is nothing for a second model to recover, and the stack will match it to within noise. Where a stack does help, it usually helps by a fraction of a point of AUC, not by a transformation of the model. That is worth real money on a large credit book and worth nothing on an internal triage tool.

The costs, counted honestly

Inference. Two boosted models means two full tree traversals per row, plus the meta-learner. Using the cost model in what batch scoring costs: if model A has 800 trees at depth 6 and model B has 600 at depth 8, the per-row comparisons go from 4,800 to 4,800 + 4,800 = 9,600, exactly doubling both the wall clock and the compute bill. For an online service, that doubling lands on p99 latency, and it lands there whether or not the second model changed the answer for that particular row.

Training. Building out-of-fold meta-features means fitting each base model k+1 times: k for the folds plus one final fit. With two base models and 5 folds, that is 12 fits per experiment instead of 1. Every hyperparameter iteration costs twelve times as much, which quietly reduces how much tuning you do.

Operations. Two models to version, two to monitor for drift, two sets of feature dependencies, and a meta-learner whose behaviour is only defined relative to the specific base models it was fitted with. Retrain one base model without refitting the meta-learner and its input distribution has shifted underneath it. This coupling is the most commonly underestimated cost, because it does not appear until the first time somebody needs to update one component in a hurry.

Debuggability. When a stacked model gives a surprising answer, the explanation has two levels, and feature attribution on the meta-learner tells you which model was trusted, not which feature mattered. If you are in a setting where a prediction has to be explained to a customer or a regulator, price that in.

Making the call

A decision rule that holds up: quantify the gain in the units the business uses, then compare it against a doubled scoring cost and a doubled operational surface.

• Worth it when a fraction of a point converts to real money at your volume, when scoring is batch rather than interactive so the latency doubling is invisible, and when you have the engineering capacity to maintain two model lineages.
• Not worth it when the gain is inside the fold-to-fold standard deviation, when you are latency-bound, when the model must be explainable per prediction, or when the base models are near-duplicates of each other.
• Try first the cheaper things that often produce a larger gain than stacking does: better features, a fixed leak, cleaning up mislabelled rows, or simply more careful tuning of the single model you already have.

One middle path worth knowing: a plain average of two models’ probabilities captures a good share of the benefit with none of the meta-learner’s coupling or leakage risk. If a simple average gets most of the way, the learned combination is buying very little for the complexity it adds.