Gradient Boosting Hyperparameter Tuning: Where to Start

A boosted-tree library exposes about thirty parameters. Three of them decide most of what you will get, one of them should almost never be tuned by search at all, and the rest are worth minutes rather than hours. The order is the tutorial.

Why the order matters more than the grid

Gradient boosting fits trees sequentially, each one on the residual errors of the ensemble so far. Every parameter you can set is controlling one of three things: how big a step each tree takes, how much a single tree can express, and how much randomness and penalty stand between the ensemble and memorising the training rows.

Those three interact, but not symmetrically. Step size and tree count are near-perfect substitutes — halve the learning rate and you need roughly twice the trees for the same fit — so tuning them jointly wastes most of a search budget exploring a ridge of equivalent configurations. Tree complexity has a genuine optimum that depends on how much interaction the signal has. Regularisation matters most when rows are few relative to columns. Search in that order and each stage starts from a sensible place instead of from noise.

One: learning rate, with early stopping

Do not put n_estimators in a search grid. Set the learning rate, set the tree count to something generously large, and let early stopping choose the count on a validation fold. That converts a two-dimensional search into a one-dimensional one, and it also gives you the right tree count for every other configuration you try afterwards rather than one shared compromise.

A learning rate of 0.1 is a reasonable place to begin on a table of tens of thousands of rows. Lower rates — 0.05, 0.03 — usually buy a small amount of accuracy at a directly proportional cost in trees and therefore in both training time and per-row scoring time. That last part is easy to forget: the tree count is the dominant term in what batch scoring costs, so a rate of 0.02 that needs 4,000 trees to win 0.2 points of AUC is a real decision, not a free one.

Set early_stopping_rounds to something like 50 for a rate of 0.1, and scale it up as the rate comes down; a slow learner improves in smaller increments and a short patience will cut it off on a plateau it would have climbed out of. The published parameter references are worth having open — the XGBoost parameter documentation and LightGBM’s parameter list both state the defaults and the valid ranges.

Two: tree complexity

This is the parameter with the most genuine variation between datasets, and the two libraries expose it differently, which is the single most common source of a bad port between them.

XGBoost grows depth-wise and is controlled by max_depth. A depth of 6 is the documented default and a tree of depth d can express interactions between at most d features along any one path. LightGBM grows leaf-wise: it repeatedly splits whichever leaf promises the biggest loss reduction, so its primary control is num_leaves and its trees are deliberately unbalanced. Setting num_leaves to 2 ** max_depth does not reproduce an XGBoost tree; it produces a far more expressive one, because a leaf-wise tree can spend all of its leaves down one branch. LightGBM documents this explicitly and recommends keeping num_leaves comfortably below that bound.

The companion parameter is the minimum data or minimum Hessian per leaf — min_child_weight in XGBoost, min_data_in_leaf in LightGBM. It is the one that actually stops a leaf being carved out for four rows. On a wide table with strong categorical features it does more for generalisation than depth does, and on small datasets it is the first thing to raise when validation and training scores diverge.

Three: sampling and regularisation

• subsample — the fraction of rows each tree sees. Values in the 0.6–0.9 range add variance between trees, which is usually what you want from an ensemble. Note that in XGBoost this samples once per boosting round.
• colsample_bytree — the fraction of columns each tree may use. This is the most useful knob on wide tables, because it forces trees to build on features that would otherwise always lose the split competition to one dominant column. It also partially decouples correlated columns, which is why it interacts with collinearity in the feature set.
• reg_lambda and reg_alpha — L2 and L1 penalties on leaf weights. Move these when the model overfits despite sensible depth and sampling, and expect logarithmic-scale search ranges rather than linear ones.
• min_split_gain / gamma — a floor on the loss reduction a split must achieve. Blunt, effective, and the fastest way to shrink a bloated model without touching depth.

A search you can run

Randomised search beats grid search here for a structural reason: grid search spends the same number of trials on the parameter that does nothing as on the one that does everything, and boosted-tree parameter importance is very uneven. The script below fixes the learning rate, lets early stopping pick the tree count, and randomises over complexity and sampling. Swap in the splitter your data needs — see choosing a cross-validation splitter before you accept any number this prints.

1. Split off a genuine holdout that the search never touches.
2. Fix learning_rate=0.1 and a large n_estimators, and confirm early stopping fires well before the cap. If it does not, the cap is too low.
3. Randomise over max_depth, min_child_weight, subsample and colsample_bytree for 40–60 trials.
4. Take the best configuration, halve the learning rate, raise the tree cap and early-stopping patience, and refit once. Keep it only if the holdout agrees.
5. Only then consider reg_lambda and reg_alpha, and only if training and validation scores are still far apart.

import numpy as np
from scipy.stats import randint, uniform
from sklearn.model_selection import RandomizedSearchCV, StratifiedKFold, train_test_split
from xgboost import XGBClassifier

X_fit, X_hold, y_fit, y_hold = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=0
)

base = XGBClassifier(
    n_estimators=3000,
    learning_rate=0.1,
    early_stopping_rounds=50,
    eval_metric="auc",
    tree_method="hist",
    random_state=0,
)

space = {
    "max_depth": randint(3, 10),
    "min_child_weight": randint(1, 30),
    "subsample": uniform(0.6, 0.4),          # 0.6 .. 1.0
    "colsample_bytree": uniform(0.5, 0.5),   # 0.5 .. 1.0
}

search = RandomizedSearchCV(
    base, space, n_iter=50, scoring="roc_auc",
    cv=StratifiedKFold(5, shuffle=True, random_state=0),
    random_state=0, n_jobs=-1, refit=True,
)
search.fit(X_fit, y_fit, eval_set=[(X_hold, y_hold)], verbose=False)

print(search.best_params_)
print("cv auc:", round(search.best_score_, 4))
print("trees kept:", search.best_estimator_.best_iteration)

Two things to read off the result rather than just the score. If the best configuration sits at the edge of a range — max_depth of 9 when the range stopped at 9 — the range was wrong and the search has not finished. And if the top ten trials are within noise of each other, further tuning is not where the remaining accuracy is; the features are.