Gradient Boosting, Explained by Building Three Trees
11 min read · updated August 4, 2026
Gradient boosting fits a tree to the errors of what you have so far, adds a fraction of it, and repeats. Every explanation says that; almost none of them shows it happening. Here are three rounds on six rows, with the split search worked, the residuals after each round, and the total squared error falling from 810 to 40.5.
The idea in one sentence
Start with a constant prediction. Compute how wrong it is on every row. Fit a small tree whose job is to predict that wrongness. Add a fraction of the tree’s output to the running prediction. Compute the new wrongness. Repeat a few hundred times.
Nothing is thrown away and nothing is refitted. Each tree is a small correction to the sum of everything before it, which is why the model is an additive one: the final prediction is a constant plus a scaled sum of tree outputs. Contrast a random forest, where every tree predicts the answer itself and the ensemble averages them.
The dataset and the first prediction
Six rows. One feature x (years of experience) and one target y (salary in thousands). The numbers are invented so the arithmetic stays clean; nothing about the method depends on them.
x y 1 30 2 33 3 39 4 51 5 57 6 60
Under squared error, the best constant prediction is the mean. (30 + 33 + 39 + 51 + 57 + 60) / 6 = 270 / 6 = 45. Call that F0 = 45. Its residuals, y minus 45, are the errors the first tree has to predict:
x y F0 r = y - F0 1 30 45 -15 2 33 45 -12 3 39 45 -6 4 51 45 6 5 57 45 12 6 60 45 15 total squared error = 225+144+36+36+144+225 = 810
Tree one: choosing the split by hand
Use stumps — trees of depth 1, one split and two leaves. That is not a simplification for the page; it is a real setting (max_depth=1) and boosted stumps fit a purely additive model with no interactions.
The split search is exhaustive: for every candidate threshold, put each row in a leaf, set each leaf’s output to the mean of the residuals in it (that is the value which minimises squared error inside the leaf), and total the leftover squared error. The winner is the threshold with the smallest total.
split left residuals right residuals SSE x < 1.5 -15 -12 -6 6 12 15 540 x < 2.5 -15 -12 -6 6 12 15 263.25 x < 3.5 -15 -12 -6 6 12 15 84 <-- best x < 4.5 -15 -12 -6 6 12 15 263.25 x < 5.5 -15 -12 -6 6 12 15 540 worked, for the winner: left mean = (-15 - 12 - 6) / 3 = -11 left SSE = (-15+11)^2 + (-12+11)^2 + (-6+11)^2 = 16 + 1 + 25 = 42 right mean = (6 + 12 + 15) / 3 = 11 right SSE = (6-11)^2 + (12-11)^2 + (15-11)^2 = 25 + 1 + 16 = 42 total = 84
So tree 1 is: if x is under 3.5 predict −11, otherwise predict +11. Now the part everyone skips. You do not add the whole tree. You add a fraction of it, the learning rate. Take ν = 0.5 here so the arithmetic stays legible; production values are 0.01 to 0.1, and the consequence of that is discussed below.
F1(x) = F0 + 0.5 * tree1(x) x y F1 r = y - F1 1 30 45 + 0.5*(-11) = 39.5 -9.5 2 33 39.5 -6.5 3 39 39.5 -0.5 4 51 45 + 0.5*(11) = 50.5 0.5 5 57 50.5 6.5 6 60 50.5 9.5 total squared error = 90.25+42.25+0.25+0.25+42.25+90.25 = 265.5 (was 810)
Tree two, on what is left
Tree two never sees y. It sees only the new residual column, and it runs exactly the same split search on it. This is the whole trick: the second problem is a fresh regression problem with a new target.
residuals to fit: -9.5 -6.5 -0.5 0.5 6.5 9.5 split SSE x < 1.5 157.2 x < 2.5 73.5 <-- best (ties with x < 4.5; first wins) x < 3.5 84.0 x < 4.5 73.5 x < 5.5 157.2 left mean = (-9.5 - 6.5) / 2 = -8 right mean = (-0.5 + 0.5 + 6.5 + 9.5) / 4 = 4
Note that the split moved. The first tree cut at 3.5; the second cuts at 2.5, because the errors that remain are concentrated at the two ends. That is boosting doing something a single tree cannot: allocating its next unit of capacity to wherever the current model is worst.
F2(x) = F1(x) + 0.5 * tree2(x) x y F2 r = y - F2 1 30 39.5 + 0.5*(-8) = 35.5 -5.5 2 33 35.5 -2.5 3 39 39.5 + 0.5*(4) = 41.5 -2.5 4 51 50.5 + 2 = 52.5 -1.5 5 57 52.5 4.5 6 60 52.5 7.5 total squared error = 30.25+6.25+6.25+2.25+20.25+56.25 = 121.5 (was 265.5)
Tree three, and the shrinking error
residuals to fit: -5.5 -2.5 -2.5 -1.5 4.5 7.5 split SSE x < 1.5 85.2 x < 2.5 73.5 x < 3.5 48.0 x < 4.5 13.5 <-- best x < 5.5 54.0 left mean = (-5.5 - 2.5 - 2.5 - 1.5) / 4 = -3 right mean = (4.5 + 7.5) / 2 = 6 F3(x) = F2(x) + 0.5 * tree3(x) x y F3 r = y - F3 1 30 35.5 - 1.5 = 34.0 -4.0 2 33 34.0 -1.0 3 39 41.5 - 1.5 = 40.0 -1.0 4 51 52.5 - 1.5 = 51.0 0.0 5 57 52.5 + 3 = 55.5 1.5 6 60 55.5 4.5 total squared error = 16+1+1+0+2.25+20.25 = 40.5 (was 121.5)
The whole run, in one column:
after total squared error F0 810.0 F1 265.5 F2 121.5 F3 40.5
Three stumps — six splits in total — took the squared error to five per cent of where it started, and the final model is nothing but 45 + 0.5·T1(x) + 0.5·T2(x) + 0.5·T3(x). Each tree is uninteresting on its own. The sum is the model.
Where the word gradient comes in
Nothing above involved a gradient, which is a fair reason to find the name confusing. The connection is one line. For squared error written as L = ½(y − F)², the derivative with respect to the prediction is
dL/dF = -(y - F) = -residual so: negative gradient = y - F = residual
Fitting the residual is fitting the negative gradient of the loss. Once that is seen, the generalisation is obvious: for any differentiable loss, compute the negative gradient of the loss with respect to the current prediction on each row, and fit the next tree to that. The algorithm never changes; only the column you hand to the tree does.
| Loss | Description |
|---|---|
| squared error | Negative gradient is y − F: the plain residual, as worked above. Regression, symmetric costs. |
| absolute error | Negative gradient is sign(y − F): +1 or −1. Every row contributes equally regardless of how wrong it is, which is why it resists outliers. |
| log loss (binary) | Predictions live in log-odds. Negative gradient is y − p where p = sigmoid(F). A row the model already calls confidently and correctly contributes almost nothing to the next tree. |
| pinball (quantile τ) | Negative gradient is τ when y > F and τ − 1 when y < F. Asymmetric on purpose, and the basis of the ordering rule in the demand-forecasting page. |
The log-loss row is worth pausing on. Boosting a classifier is identical to what happened above, with the running prediction held in log-odds and the residual column being y − sigmoid(F). That is also the reason boosted classifiers are usually badly calibrated at the extremes: the optimisation keeps pushing confident rows further out because doing so still reduces log loss.
What each hyperparameter does to this
- learning_rate (ν). The fraction of each tree that is added. At ν = 0.5 the error fell fast in three rounds. At ν = 0.05 each round moves a tenth as far, so you need roughly ten times as many trees for the same fit — and the fit generalises better, because no single tree gets to commit the ensemble to a split it found in noise. Learning rate and tree count trade off almost exactly; tune one and set the other by early stopping.
- max_depth / num_leaves. Depth 1 gives an additive model with no interactions, as above. Depth 2 lets one tree express a two-way interaction, depth 3 a three-way, and so on. Most tabular work sits at depth 3 to 8, and depth is the fastest way to overfit.
- subsample / colsample. Fit each tree on a random fraction of rows or columns. This is where boosting borrows from bagging: it decorrelates consecutive trees and acts as a regulariser.
- min_child_weight / min_samples_leaf. Refuse a split that would leave a leaf with too little data. With six rows the splits above would be blocked by any sane setting, which is the whole reason boosting needs thousands of rows to behave.
- n_estimators with early stopping. Set it high and stop when a validation score stops improving. The one number in boosting you should never tune by hand.
Reproducing the table in code
Scikit-learn’s GradientBoostingRegressor initialises with the mean under squared error, so with three stumps and a learning rate of 0.5 it follows the same path.
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor
X = np.array([[1], [2], [3], [4], [5], [6]])
y = np.array([30, 33, 39, 51, 57, 60], dtype=float)
for n in range(0, 4):
if n == 0:
pred = np.full_like(y, y.mean())
else:
m = GradientBoostingRegressor(
n_estimators=n, learning_rate=0.5, max_depth=1,
loss="squared_error", random_state=0,
).fit(X, y)
pred = m.predict(X)
sse = float(((y - pred) ** 2).sum())
print(f"after {n} trees: sse = {sse:.1f} preds = {np.round(pred, 2)}")The second and third rounds contain a tie in the split search, and implementations are free to break it either way; if yours picks x < 4.5 rather than x < 2.5 in round two the later numbers will differ from the table while the error still falls monotonically. That is worth seeing rather than hiding, because it is the same non-determinism that makes two boosted models trained on the same data disagree on individual rows.
For why this family still beats neural networks on tables rather than how it works, see the ranking argument with the published evidence, and for the alternative ensembling philosophy see bagging.