Class Imbalance: Techniques That Actually Help

The standard advice on imbalance is a list — oversample, undersample, SMOTE, class weights, move the threshold — presented as alternatives to be tried. Three of those are the same operation applied at different points in the pipeline, and seeing why collapses the list.

What imbalance actually breaks

Not the model, usually. A classifier trained with cross-entropy on a 1% positive class can be perfectly well calibrated: it outputs low probabilities because the event is rare, and that is correct behaviour. Nothing is broken about the probabilities.

What breaks is everything downstream of them. Taking the argmax — or equivalently thresholding at 0.5 — predicts the majority class for almost every input, because a well-calibrated posterior rarely exceeds 0.5 when the prior is 0.01. And accuracy, computed on those predictions, reports 99% while the model has caught nothing. Both failures are in the decision rule and the metric, not in the fit.

Which reframes the problem usefully: fix the metric first, then the threshold, and only then consider touching the data. Teams that reach for SMOTE before they have looked at a precision-recall curve are usually solving a metric problem with a data intervention.

The correction, derived

Suppose you resample so the training prior is π’ instead of the true prior π. What does the model learn?

By Bayes, the posterior a model fits is proportional to the likelihood times the prior: p(y|x) ∝ p(x|y)·p(y). Resampling changes which examples you draw, but it does not change p(x|y) — the appearance of a fraudulent transaction is the same whether you showed the model ten of them or ten thousand. Only the prior changes. So the model fits p'(y|x) ∝ p(x|y)·π', and taking the ratio to the true posterior:

p(y|x)     π
──────  =  ──      (up to the normalising constant)
p'(y|x)    π'

in log-odds / logit space, that is an additive shift:

    z_corrected = z' + log(π / π')

A constant added to the logit. Which is to say: oversampling the minority class by a factor r shifts its logit by log r, and you could have obtained the identical decision rule by leaving the data alone and lowering the threshold by log r in logit space. Resampling and threshold moving are the same shift, applied before training and after it.

This is not a curiosity. King and Zeng (2001) derived the prior correction for rare-events logistic regression on exactly this basis, and Menon and colleagues (ICLR 2021, “Long-tail learning via logit adjustment”) develop the same additive logit shift as a principled alternative to resampling for long-tailed classification. The published work agrees that the shift is the content; where you apply it is an engineering choice.

And there is a strong practical reason to prefer applying it last. Resampling changes the training distribution, so the model’s outputs are no longer calibrated for your traffic, and every probability you report downstream is wrong until corrected. Thresholding leaves calibration intact and takes one line. It also costs nothing to change your mind about later, which resampling does not — that requires retraining.

Why class weights are duplication

The third member of the family. Multiply each example’s loss by a class-dependent weight w_c:

L = Σᵢ w_{y_i} · CE(pᵢ, yᵢ)

∂L/∂θ = Σᵢ w_{y_i} · ∂CE(pᵢ, yᵢ)/∂θ

Each example’s gradient contribution is scaled by its class’s weight — which is exactly what would happen if you had included that example w_c times. In expectation, weighting class c by w_c is oversampling it w_c times, and therefore, by the previous section, a logit shift of log w_c. One family, three implementations.

The differences that remain are practical rather than statistical. Weighting keeps the dataset intact and costs nothing in wall-clock time; oversampling multiplies epoch length; undersampling throws away majority data, which is real information loss and only sensible when the majority class is enormous and redundant.

The one case resampling is not optional

There is a regime where the equivalence above stops being the whole story, and it is a batch-composition problem rather than a distributional one. If positives are rare enough, most minibatches contain none at all, and a batch with no positives contributes nothing to the decision boundary you care about.

P(at least one positive in a batch) = 1 − (1 − π)^B

π = 0.01,  B = 32   →  1 − 0.99³²   = 27.5%
π = 0.001, B = 32   →  1 − 0.999³²  =  3.2%
π = 0.001, B = 256  →  1 − 0.999²⁵⁶ = 22.5%

At one in a thousand with a batch of 32, 97% of your updates see no positive example. Here, oversampling or a stratified batch sampler is fixing something a threshold cannot: it is making sure the gradient signal exists at all. Raising the batch size does the same job, at a different cost.

What to do, in order

• Replace accuracy. Precision, recall, a precision-recall curve, and the prevalence stated beside it. Most “imbalance problems” are visible only after this step and some of them dissolve at it.
• Tune the threshold on validation, against costs. The single highest-value intervention, and it requires no retraining. Sweep, compute expected cost, pick the minimum.
• Check batch composition. Use the formula above. If most batches are empty of positives, use a stratified sampler or a larger batch.
• Then consider weighting. Cheap, reversible, equivalent to resampling without the epoch-length cost. Remember it decalibrates your probabilities in the same way, so correct them if you report them.
• SMOTE last, and sceptically. Chawla et al. (2002) interpolate between a minority example and its neighbours to synthesise new ones. It cannot create information that was not in the sample, it interpolates in a space where the interpolation may be meaningless (halfway between two categorical encodings is nothing), and applying it before the split leaks synthetic relatives of training rows into your test set. If you use it, use it inside the fold.
• More minority data beats all of the above. Fifty more real positives are usually worth more than any technique on this list, and that is worth saying to whoever owns the labelling budget.