Random Forests and Bagging: Where the Variance Goes
10 min read · updated August 4, 2026
A random forest averages many deep trees, each fitted to a different random view of the data. The averaging cannot make the individual trees better, and it cannot remove bias. What it removes is variance, and exactly how much is derivable in two lines of arithmetic that also explain why adding the thousandth tree does nothing.
Two ideas, not one
Bagging is the first: fit each tree on a bootstrap sample — n rows drawn with replacement from the n you have — so each tree sees a different dataset and makes different mistakes.
Random feature selection is the second, and it is what makes a forest a forest rather than just bagged trees. At every split, the tree is allowed to consider only a random subset of the columns, typically the square root of the total for classification or a third for regression. Without it, one dominant feature would sit at the root of every tree and all the trees would look alike. The next section shows precisely why looking alike is the problem.
Individual trees are grown deep and left unpruned. A single deep tree is a low-bias, high-variance estimator — it fits the training data almost exactly and moves wildly when the data moves. That is the ideal ingredient for averaging, and the opposite of the shallow stumps boosting uses.
The averaging identity, worked
Take one held-out row whose true value is 41. Five trees predict it:
tree prediction error vs truth 41 squared error 1 30 -11 121 2 45 +4 16 3 38 -3 9 4 52 +11 121 5 35 -6 36 mean squared error of an individual tree = (121+16+9+121+36)/5 = 303/5 = 60.6 ensemble prediction = (30+45+38+52+35)/5 = 200/5 = 40 ensemble squared error = (40 - 41)^2 = 1.0
The average tree is off by a squared error of 60.6. The average of the trees is off by 1.0. That is not a coincidence and it is not luck on this example; it is an identity. For squared error, for any ensemble of any predictors, on any point:
mean individual error = ensemble error + mean spread about the ensemble
check on the numbers above:
spread = [(30-40)^2 + (45-40)^2 + (38-40)^2 + (52-40)^2 + (35-40)^2] / 5
= (100 + 25 + 4 + 144 + 25) / 5
= 298 / 5
= 59.6
1.0 + 59.6 = 60.6 <-- equals the mean individual error exactlyRead the identity right to left. The ensemble error is the mean individual error minus the spread, and the spread cannot be negative. So the ensemble is never worse than the average member, and it is better by exactly how much its members disagree. Disagreement is not a defect to be minimised. It is the mechanism.
This is also why an ensemble of five identical trees is worth nothing: the spread term is zero and the ensemble error equals the individual error. Every part of a forest’s design — the bootstrap, the feature subsetting, the lack of pruning — exists to make that term large.
Why more trees stop helping
The identity above holds for one point. Averaged over datasets, the variance of a mean of B estimators, each with variance σ² and pairwise correlation ρ, is:
Var(mean of B trees) = ρ·σ² + (1 - ρ)·σ² / B with σ² = 1 and ρ = 0.6: B = 1 0.6 + 0.4/1 = 1.000 B = 10 0.6 + 0.4/10 = 0.640 B = 50 0.6 + 0.4/50 = 0.608 B = 100 0.6 + 0.4/100 = 0.604 B = 1000 0.6 + 0.4/1000 = 0.6004 B -> inf 0.600 <-- the floor
The second term vanishes with more trees. The first does not: it is ρ·σ², and no number of trees touches it. That single line explains three things that otherwise look like folklore.
- Why 100 to 500 trees is usually enough. By B = 100 the reducible term is down to 0.4% of σ². Going to 1,000 buys 0.06% and costs ten times the inference latency.
- Why feature subsampling is the important hyperparameter. The only lever on the floor is ρ. Restricting each split to a random subset of columns is a direct attack on the correlation between trees, at the cost of making each tree slightly worse (σ² rises a little). That trade is the whole design of a random forest, and it is why
max_featuresmatters more thann_estimators. - Why a forest cannot fix a biased model. The formula is about variance. If every tree is systematically wrong in the same direction — because a feature is missing, or the label is defined wrongly — averaging preserves the error perfectly. Bagging is not a remedy for a broken dataset.
Out-of-bag: a free validation set
A bootstrap sample of n rows drawn with replacement leaves some rows out. How many is derivable. The chance one particular row is missed on one draw is (1 − 1/n), and there are n draws:
P(row is out of bag) = (1 - 1/n)^n -> e^-1 = 0.3679 as n grows n = 10 0.349 n = 100 0.366 n = 1000 0.368
So roughly 37% of rows are unused by any given tree, and each row is out-of-bag for roughly 37% of the trees. Predicting each row using only the trees that never saw it gives an honest held-out estimate at no cost in data — oob_score=True in scikit-learn’s RandomForestClassifier.
Forest against boosting
| Property | Description |
|---|---|
| what each tree fits | Forest: the target itself, on a resampled dataset. Boosting: the current residual, on all the data. This is the whole difference; everything below follows from it. |
| tree depth | Forest: deep, unpruned, low bias. Boosting: shallow, typically 3–8, because depth is added by summing trees rather than by growing them. |
| effect of more trees | Forest: converges to a floor and cannot overfit by adding trees. Boosting: keeps reducing training error and will overfit, which is why it needs early stopping. |
| parallelism | Forest: every tree is independent, so training parallelises perfectly. Boosting: tree k+1 depends on tree k, so only the split search within a tree parallelises. |
| typical accuracy on tables | Boosting usually ahead when tuned, forest usually ahead when nothing is tuned. A forest with default settings is a genuinely strong baseline; boosting with default settings often is not. |
| sensitivity to noisy labels | Forest more robust: a mislabelled row affects the trees that sampled it. Boosting chases it, because a badly-fit row produces a large gradient that the next tree is built to attack. |
The last row is the practical reason to reach for a forest first on a dataset you do not trust yet. Boosting will happily spend a hundred trees fitting twenty mislabelled rows.
The importance trap
The feature_importances_ attribute on a fitted forest is impurity-based: it totals how much each feature reduced impurity across all splits. It has a documented bias towards high-cardinality and continuous features, because a column with a thousand distinct values offers a thousand candidate thresholds and will win splits by chance alone. Add a column of random floats to any dataset and it will not rank last.
Permutation importance, computed on held-out data, is the alternative: shuffle one column and measure how much the score drops. It answers a question you actually asked, and scikit-learn ships it as sklearn.inspection.permutation_importance. It has its own failure mode with correlated features — shuffle one of two duplicate columns and neither looks important, because the other covers for it. What an explanation is and is not evidence of covers this properly.
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
rf = RandomForestClassifier(
n_estimators=300, max_features="sqrt",
min_samples_leaf=5, oob_score=True, n_jobs=-1, random_state=0,
).fit(X_tr, y_tr)
print("oob:", rf.oob_score_)
r = permutation_importance(rf, X_te, y_te, n_repeats=20,
scoring="roc_auc", random_state=0)
for i in r.importances_mean.argsort()[::-1][:10]:
print(f"{X_te.columns[i]:30s} {r.importances_mean[i]:+.4f}"
f" +/- {r.importances_std[i]:.4f}")Print the standard deviation next to the mean, as above. A feature whose importance is 0.004 with a spread of 0.006 has not been shown to matter, and reporting it without the spread is how importance tables become fiction.
Settings that matter
- n_estimators. Set it to 300–500 and stop thinking about it. The variance formula says the returns beyond that are negligible, and unlike boosting, more trees cannot hurt accuracy.
- max_features. The real lever, because it is the only one that moves ρ. Try
"sqrt",0.3and1.0; on datasets with many weak, correlated features the lower values usually win by more than any other single change. - min_samples_leaf. The main capacity control. Leave at 1 for classification with clean labels; raise to 5 or 20 when labels are noisy or n is small, which shortens training and often improves the held-out score.
- class_weight. For imbalanced problems, set
"balanced_subsample"rather than resampling by hand. Then ignore the resulting probabilities until you have read the calibration page, because reweighting shifts them and the class-imbalance toolkit trades one problem for another. - n_jobs=-1. Free, because the trees are independent.
A forest is a good default and a bad ceiling. When the marginal point of AUC matters, it is usually boosting that gets it — and when nothing matters more than a reliable first number in an afternoon, this is the model to fit.