Numeric Feature Scaling: Standardisation, Normalisation and When Each Distorts a Distribution

Standardisation and min-max scaling are both linear maps. That single fact predicts everything they do and everything they cannot do: they move and stretch a distribution, and they never change its shape.

One skewed column

Ten customers’ annual spend, which is the canonical right-skewed business quantity:

x = [120, 150, 180, 200, 210, 240, 260, 300, 340, 8000]

n        = 10
sum      = 10,000
mean     = 1,000.0
median   = 225.0
min      = 120
max      = 8,000

sum of squared deviations from the mean = 55,255,000
variance (population, /n)               =  5,525,500
std      = 2,350.6
skew     = strongly right; nine of ten values sit below the mean

Note the mean of 1,000 against a median of 225 before doing anything else. The mean is above every value except one. Any method that centres on the mean is centring on a point where no customer is.

Standardisation

Subtract the mean, divide by the standard deviation. The result has mean 0 and standard deviation 1 by construction:

z = (x - 1000) / 2350.6

  120  →  -0.374
  150  →  -0.361
  180  →  -0.349
  200  →  -0.340
  210  →  -0.336
  240  →  -0.323
  260  →  -0.315
  300  →  -0.298
  340  →  -0.281
 8000  →   2.978

nine values span  -0.374 to -0.281   (a range of 0.093)
one value sits at  2.978
that one value is 32 times further from the pack than the pack is wide

The transform did what it promised and the column is still hopeless. Nine of ten customers now occupy a band of width 0.093 while a single outlier sits three units away. A distance-based model sees ten points of which nine are effectively coincident. The reason is circular and worth stating: the outlier inflated the standard deviation to 2,350, and the standard deviation is the divisor, so the outlier set the scale that then crushed everything else.

This is why scikit-learn’s preprocessing guide states plainly that if your data contains many outliers, scaling using the mean and variance is likely to not work very well, and recommends RobustScaler as a drop-in replacement using more robust estimates of centre and range. Substituting the median (225) and the interquartile range for the mean and standard deviation gives the nine ordinary customers a usable spread and leaves the outlier far out where it belongs — which is the correct behaviour, because it is genuinely far out.

Min-max normalisation

Map the observed range onto [0, 1] by subtracting the minimum and dividing by the range:

x' = (x - 120) / (8000 - 120) = (x - 120) / 7880

  120  →  0.0000
  150  →  0.0038
  180  →  0.0076
  200  →  0.0102
  210  →  0.0114
  240  →  0.0152
  260  →  0.0178
  300  →  0.0228
  340  →  0.0279
 8000  →  1.0000

nine values occupy 0.0000 to 0.0279 — 2.8% of the output range
97.2% of the range holds one point

Worse, and in a specific way: the entire ordinary population is compressed into under three per cent of the axis. In float32 there is still plenty of numerical precision, so nothing is lost arithmetically — but every distance computed on this column is now dominated by whether a row is the outlier or not, and any other column scaled the same way will overwhelm it in a Euclidean distance. That is exactly the effect worked through numerically in tabular row embeddings, where the same two customers change their similarity ranking purely on the choice of scaler.

Min-max has a second property that standardisation does not: it is unbounded on new data. The parameters are fitted on the training set, so a test row with spend of 12,000 maps to 1.51, outside the range the model was trained to expect. Clipping hides it, which turns an out-of-range input into a silently in-range one. Standardisation produces a large z-score instead, which at least remains interpretable.

What neither of them fixes

Compute the skew before and after either transform and it is identical. Both maps are of the form ax + b, and skewness is invariant under a positive affine transformation, so scaling cannot change it. The distribution before was one enormous value and nine small ones; afterwards it is one enormous value and nine small ones on a different axis.

Say precisely what scaling therefore is and is not for. It is for putting features on comparable scales so that an optimiser converges, a regulariser penalises coefficients fairly, and a distance metric is not dominated by whichever column happens to be measured in a large unit. It is not for making a distribution well-behaved. Those are different problems requiring different transforms, and conflating them is why people apply StandardScaler to a log-normal column and wonder why the model still struggles.

Transforms that change the shape

To change the shape you need something non-linear. Three options, in increasing aggressiveness.

• A log transform. For strictly positive right-skewed quantities, log(x) or log1p(x) is the simplest thing that works. On the worked column, log10 maps 120 to 2.08 and 8,000 to 3.90 — the outlier is now less than twice the smallest value rather than 67 times it, and the nine ordinary customers spread across 2.08 to 2.53 with real resolution between them. It also has a meaning: modelling log(spend) makes the model additive in log space, which is multiplicative in the original space, and for spend that is usually the more natural claim.
• A power transform. Box-Cox and Yeo-Johnson fit an exponent to make the result as close to Gaussian as possible. scikit-learn’s documentation is explicit that Box-Cox can only be applied to strictly positive data, so Yeo-Johnson is the one to reach for when a column contains zeros or negatives.
• A quantile transform. Replace each value with its rank, mapped to a uniform or normal distribution. It is the most forceful option and the docs note the trade directly: a rank transformation is less influenced by outliers than scaling methods, but it distorts correlations and distances within and across features. Ranks discard all information about magnitude — after a quantile transform, a customer spending 8,000 is simply the largest, not 24 times the median — and if the magnitude mattered, you have thrown away the signal.

Binning is the fourth option and a different kind of trade, since it converts a continuous variable to an ordered categorical one; the trade-offs are in variable binning trade-offs.

Which models need it at all

Scaling is mandatory for anything that computes a distance or a dot product, or that optimises with gradient descent: k-nearest neighbours, k-means, support vector machines, principal component analysis, regularised linear models, and neural networks. In the regularised case the reason is often missed — an L1 or L2 penalty applies equally to every coefficient, so a feature measured in a small unit gets a large coefficient and is penalised more for no reason connected to its usefulness.

Decision trees and their ensembles do not need it, and this is a genuine property rather than a tolerance. A tree splits on a threshold, and any monotonic transform preserves the ordering of the values, so the same partition is available before and after. Applying a scaler before a gradient-boosted model changes nothing but the numbers in the log — one of the practical conveniences behind why gradient boosting is still the default on tabular data.

Two rules regardless. Fit the scaler on the training fold only; a scaler fitted on the full dataset has carried the test set’s mean and range into training, and the resulting score is not an estimate of performance on unseen data. And persist the fitted parameters with the model, because inference must apply the identical transform — recomputing the mean on a batch at serving time puts every prediction in a slightly different space from the one the model learned.