Balancing a Dataset Without Throwing Data Away

A dataset that is 97% one class provokes an immediate instinct to rebalance it. Before acting on that instinct it is worth asking what the imbalance is actually breaking, because in a large number of cases the answer is “nothing, except the metric you chose”.

Usually the wrong question

Class imbalance is not a defect in itself. If 3% of transactions are fraudulent, then a dataset that is 3% fraud is a correct sample of reality, and reality is what the model will meet in production. Rebalancing it to 50/50 does not fix a problem with the data — it introduces a mismatch between training and deployment.

What imbalance genuinely breaks is narrower than people assume:

• Accuracy as a metric. Always predicting the majority class scores 97%. This is a problem with accuracy, and the fix is to use precision, recall, PR-AUC or a cost-weighted metric — not to change the data.
• Absolute minority counts, at the extreme. Forty positive examples is a problem. Forty thousand positives out of four million is not, despite being a worse ratio. The count matters; the ratio mostly does not.
• Optimisation dynamics with small batches. If a batch of 32 usually contains no positive example, gradients are dominated by the majority class. Real, and addressed by batch composition rather than by resampling the whole dataset.
• The default decision threshold. A classifier thresholded at 0.5 will rarely predict a rare class. This is the most common complaint and it is fixed by moving the threshold, which requires changing nothing about the data at all.

Notice that two of the four are metric problems and one is a threshold problem. Only the small-absolute-count case is genuinely a data problem, and it has a different fix from the others.

The four options

The genuinely different option is the fifth: collect more minority data. When the problem is forty positive examples, no reweighting scheme creates information that is not there. Targeted acquisition — searching the unlabelled pool specifically for likely positives and labelling those — is a much better use of a budget than any resampling scheme, and it is the only option that raises the ceiling.

What resampling does to your probabilities

This is the consequence most often missed, and it is the reason to be conservative about rebalancing whenever the model’s output is a probability rather than a decision.

A classifier trained on a resampled dataset learns the prior of the resampled data. Rebalance a 3% positive rate to 50% and the model’s outputs are calibrated to a world where half of all cases are positive. Its 0.6 does not mean 60%. Every downstream consumer that treats the score as a probability — an expected-value calculation, a triage queue with a cost threshold, a risk display shown to a person — is now systematically overestimating risk, and nothing in the metrics will say so, because ranking metrics like AUC are invariant to exactly this distortion.

Van den Goorbergh and colleagues made this concrete for clinical risk prediction in The harm of class imbalance corrections for risk prediction models (JAMIA, 2022): imbalance corrections did not improve discrimination and did produce strongly miscalibrated probability estimates. In a setting where the number itself is the product, that is not a side effect — that is the model being wrong.

If you must rebalance and you need calibrated outputs, correct the prior back afterwards. For a resampling factor applied to the positive class, the log-odds shift is known and constant, so the correction is a fixed offset:

import numpy as np

def correct_prior(p_resampled, pi_train, pi_true):
    """Undo the prior shift introduced by resampling.

    pi_train — positive rate the model was TRAINED on (e.g. 0.50)
    pi_true  — positive rate in DEPLOYMENT (e.g. 0.03)

    The resampling multiplies the odds by a constant factor, so subtracting
    the corresponding constant from the log-odds recovers the true scale."""
    eps  = 1e-12
    odds = p_resampled / np.clip(1 - p_resampled, eps, None)
    shift = (pi_true / (1 - pi_true)) / (pi_train / (1 - pi_train))
    return (odds * shift) / (1 + odds * shift)

# A "0.60" from a model trained at 50/50, deployed where the real rate is 3%:
print(correct_prior(0.60, pi_train=0.50, pi_true=0.03))   # ≈ 0.044

# Then VERIFY on a natural-distribution holdout, because the analytic
# correction assumes the only thing that changed was the prior.

The last comment is the important one. The analytic correction assumes resampling changed nothing but the base rate, which is not exactly true, so check the result against a reliability curve computed on a holdout with the natural class distribution. If the corrected probabilities do not track observed frequencies, fit a calibration map on that holdout instead.

The decision rule

• Do you need probabilities or decisions? If probabilities: do not resample. Use the natural distribution, use class weights sparingly if at all, and check calibration explicitly.
• How many minority examples do you have in absolute terms? Under a few hundred: the problem is data volume, and acquisition or targeted generation is the answer. Thousands: the ratio is probably not your problem.
• Is the metric the actual complaint? If the symptom is “97% accuracy but it never predicts fraud”, change the metric and the threshold first. That takes an hour and frequently ends the discussion.
• Is the majority class redundant? Run a duplicate check before undersampling. Removing near-duplicate majority rows is deduplication and is unambiguously good; removing distinct majority rows is discarding evidence.
• Try nothing first, and measure. Train on the natural distribution, plot the precision–recall curve, choose the operating point. That baseline is required before any rebalancing scheme can be said to have helped — and it wins often enough to be worth the twenty minutes.

The mistake that invalidates the measurement

Whatever you decide, one rule is absolute: never rebalance the validation or test set. They exist to estimate performance in deployment, and deployment has the natural distribution. A test set rebalanced to 50/50 reports a precision that has no relationship to the precision you will observe, and it errs optimistically — the majority-class false positives that will swamp you in production have been sampled away.

Two corollaries. Split before resampling, never after, or duplicated minority rows land on both sides and the score measures memorisation — the same leak the augmentation page warns about, arriving from a different direction. And stratify the split so each fold contains a proportional share of the rare class: with a 1% positive rate and a small test set, an unstratified random split can produce a fold with almost no positives at all, and every number computed from it is noise.