Detecting Multicollinearity Before It Wrecks a Model

Multicollinearity is one of the most over-applied warnings in applied modelling. It genuinely destroys some things and is genuinely harmless to others, and the difference is not a matter of degree — it depends on what you are asking the model for.

What it breaks, and what it does not

Multicollinearity means one predictor is well approximated by a linear combination of the others. In the extreme case — perfect collinearity — the design matrix is singular and there is no unique least-squares solution at all; infinitely many coefficient vectors give exactly the same fit. Short of that, the solution is unique but poorly determined, and that is the whole of the problem.

It breaks coefficient estimates. The standard error of each coefficient inflates, so the estimates swing wildly with small changes in the sample. Coefficients flip sign. A predictor everyone knows increases risk gets a negative coefficient, because its collinear partner absorbed the effect and this one is now estimating a small residual difference.

It breaks significance testing. Two collinear predictors can each be individually non-significant while a joint F test on both is strongly significant. Neither is redundant; the data just cannot say which one is doing the work.

It breaks feature importance of every attribution method that assigns credit to individual columns. If two columns carry the same information, any split of the credit between them is defensible, and different methods and different random seeds will split it differently.

It does not break prediction on the same distribution. This is the part routinely got wrong. If you only need y-hat, and the future data has the same correlation structure as the training data, collinearity costs you essentially nothing — the fitted surface is fine even though the individual coefficients are unstable. The caveat is load-bearing: if the correlation breaks down in production, a model that leaned on a large positive coefficient cancelled by a large negative one produces wild predictions. Collinearity converts a distribution shift into a much larger error than it would otherwise be.

VIF: what the number is

The variance inflation factor for predictor j is computed by regressing that predictor on all the others and taking

VIF_j = 1 / (1 - R2_j)

where R2_j is the coefficient of determination of the auxiliary regression
of feature j on every other feature (not on the target).

The name is literal: VIF is the factor by which the variance of that coefficient’s estimate is multiplied, relative to a hypothetical fit where the predictor was orthogonal to all the others. A VIF of 9 means the standard error is three times larger than it would be — the square root, because variance is squared error.

Read off the arithmetic: R² of 0.50 gives VIF 2, R² of 0.80 gives VIF 5, R² of 0.90 gives VIF 10, R² of 0.99 gives VIF 100. The familiar thresholds of 5 and 10 are conventions, not results; they correspond to “80% of this predictor is explained by the others” and “90%”. Nothing changes discontinuously at either point, and treating 4.9 as fine and 5.1 as a crisis is superstition.

A worked feature set

Six features on a property price model: floor_area_m2, floor_area_sqft (a unit conversion of the first, added by a second pipeline), bedrooms, bathrooms, year_built, distance_to_station_km. Two of these are obviously collinear and one pair is mildly so.

import numpy as np, pandas as pd
import statsmodels.api as sm
from statsmodels.stats.outliers_influence import variance_inflation_factor

def vif_table(X: pd.DataFrame) -> pd.Series:
    Xc = sm.add_constant(X)                      # required; see note above
    vifs = [variance_inflation_factor(Xc.values, i) for i in range(Xc.shape[1])]
    return pd.Series(vifs, index=Xc.columns).drop("const").sort_values(ascending=False)

print(vif_table(X).round(2))

# floor_area_sqft    inf       <- exact linear function of floor_area_m2
# floor_area_m2      inf
# bedrooms          3.41
# bathrooms         3.02
# year_built        1.18
# distance_to_station_km  1.09

The two area columns return infinity, or an enormous finite number limited by floating-point rounding, because R² of the auxiliary regression is 1.0 and the denominator is zero. That is the signature of an exact duplicate, and it should be handled by a redundancy audit before VIF is ever computed — see duplicate and redundant column detection. Drop floor_area_sqft and recompute; the remaining VIFs fall into the 1–4 range and nothing needs further action.

The bedrooms and bathrooms pair at around 3 is the instructive case. They are correlated because larger properties have more of both, and each still contributes something the other does not — a three-bed one-bath and a three-bed three-bath are different properties. A VIF of 3 is not a reason to delete a real predictor. If you want to know whether the model needs both, ask the model, not the VIF.

What it does to a tree ensemble

Trees have no coefficients, so the variance-inflation story does not apply. Two things happen instead.

First, importance is split arbitrarily between collinear columns. At each node the split finder picks whichever of two near-identical columns has the marginally better gain on that node’s rows, which is close to a coin flip. Gain-based importance for each column ends up near half of what one column alone would have scored, and permutation importance for each ends up near zero, because permuting one leaves the other intact and the model barely notices. Both readings are artefacts. The fix for interpretation is to group collinear features and permute the group together, which is the approach scikit-learn documents in its permutation importance with multicollinear features example.

Second, colsample_bytree interacts with it. Sampling half the columns per tree means some trees get one member of a collinear pair and some get the other, which spreads the ensemble’s reliance across both and is mildly stabilising. It also means the same feature set can produce different importance rankings on different runs, which surprises people who expected a deterministic answer.

What to do about it

• Prediction only, stable correlation structure: nothing. Compute VIF for your own information and move on.
• You need interpretable coefficients: drop or combine the collinear group. Combining is often better — a sum, a ratio, or a first principal component of the group keeps the information and produces one coefficient with a meaning. Scale the columns first if you are taking a principal component, since PCA is not scale-invariant.
• You need coefficients but not exclusivity: ridge regression. The L2 penalty makes the problem well-posed again by shrinking the collinear coefficients toward each other rather than letting one swing positive and the other negative. This is precisely the case ridge was invented for.
• Exact duplicates and unit conversions: delete, always and without analysis. These are data-pipeline defects wearing a statistical costume.

And a caution about automated pruning: a loop that drops the highest-VIF column and recomputes until everything is under 5 will happily delete a predictor you needed, because VIF knows nothing about the target. If the goal is a smaller feature set rather than stable coefficients, use a method that looks at the outcome — automated feature selection is the right tool for that job and VIF is not.