Detecting a Mislabeled Row in a Training Set

A model that disagrees confidently with a label is making a claim about the label. Sometimes it is wrong and the example is just hard. Often enough it is right, and the row is mislabelled — and separating those two cases is the entire skill.

Why label noise is worth hunting

Label errors are more common than most teams assume, including in datasets that are treated as ground truth. Northcutt, Athalye and Mueller estimated errors in the test sets of ten widely used machine learning benchmarks in “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks”, presented at the NeurIPS 2021 datasets and benchmarks track, and put the average error rate across those ten test sets at about 3.4%. If curated academic benchmarks carry that much noise, a label column assembled from an operational system carries more.

The damage is not uniform, and this is the part that decides whether it is worth your time. Noise in the training set costs accuracy but a large ensemble absorbs a surprising amount of it. Noise in the evaluation set is worse, because it corrupts every decision you make on the basis of a score: model selection, threshold setting, and the go/no-go call. A 3% error rate in the test labels puts a hard floor under the error you can measure and can reverse the ranking of two models that differ by less than that.

Noise is also rarely random. Labels applied by a rushed reviewer, by a heuristic rule, or by a system that defaults a field on timeout are wrong in a patterned way — concentrated in one class, one time window, or one source. That pattern makes them findable, and it also means deleting them changes the class balance.

The signal: out-of-fold confidence

A model’s prediction on a row it was trained on tells you nothing; it can memorise the label, error and all. The usable signal comes from out-of-fold predictions: for each row, the predicted probability from a model fitted on folds that excluded it. Then the suspicious rows are those where the predicted probability of the recorded label is low.

Two derived statistics, and they behave differently:

• Self-confidence: the out-of-fold predicted probability assigned to the label the row actually carries. Low is suspicious. This is the simplest and most robust ranking.
• Margin: the probability of the recorded label minus the highest probability among the other classes. A negative margin means the model preferred a different class. The size of the negative margin orders the candidates by how strongly the model disagrees.

The second signal is fold disagreement. Repeat the cross-validation with several different random splits and record how often each row is predicted as something other than its label. A row that every split contradicts is a stronger candidate than one that a single unlucky fold contradicts, and this distinction matters because the whole procedure is otherwise vulnerable to noise in the model rather than in the label.

Confident learning and class-conditional noise

The refinement that makes this into a method rather than a heuristic is to stop using a single global threshold. Northcutt, Jiang and Chuang set it out in “Confident Learning: Estimating Uncertainty in Dataset Labels”, published in the Journal of Artificial Intelligence Research in 2021. The idea: for each class, compute a per-class threshold as the average out-of-fold predicted probability of that class among rows labelled with it. Then count a row labelled i as evidence of the mislabelling i→j when the model’s probability for class j exceeds class j’s own threshold.

That per-class calibration is what makes it work on imbalanced and unevenly-difficult problems. A single global cut-off of, say, 0.5 flags almost every row of a rare class, because a model on a 2% base rate rarely predicts above 0.5 for anything. The per-class threshold adapts to how confident the model typically is about that class, so a rare-class row is compared against rare-class confidence.

The output is a matrix of estimated label transitions, which is more informative than a flat list of suspect rows: if 80% of your estimated errors are “labelled negative, probably positive”, you have a systematic under-reporting problem in one direction, not scattered noise. The cleanlab library implements this and is the reference implementation of the paper.

A worked labelled example

Start with a clean synthetic table, flip 3% of the labels at random, and check how many of the flipped rows the method surfaces. Because the injected errors are known, this measures the detector rather than asserting anything about it.

import numpy as np, pandas as pd
from sklearn.datasets import make_classification
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.model_selection import StratifiedKFold, cross_val_predict

X, y_true = make_classification(n_samples=20_000, n_features=20, n_informative=8,
                                class_sep=1.2, random_state=0)

rng = np.random.default_rng(0)
flip = rng.choice(len(y_true), size=int(0.03 * len(y_true)), replace=False)
y = y_true.copy()
y[flip] = 1 - y[flip]                                  # inject known label noise

def oof_proba(seed):
    cv = StratifiedKFold(5, shuffle=True, random_state=seed)
    return cross_val_predict(
        HistGradientBoostingClassifier(random_state=seed),
        X, y, cv=cv, method="predict_proba",
    )

runs = [oof_proba(s) for s in range(5)]
p = np.mean(runs, axis=0)                              # average over 5 split seeds

self_conf = p[np.arange(len(y)), y]                    # prob of the recorded label
other     = p[np.arange(len(y)), 1 - y]
margin    = self_conf - other
disagree  = np.mean([r[np.arange(len(y)), 1 - y] > 0.5 for r in runs], axis=0)

# per-class threshold, as in confident learning
thresholds = np.array([p[y == c, c].mean() for c in (0, 1)])
flagged = other > thresholds[1 - y]

report = pd.DataFrame({"self_conf": self_conf, "margin": margin,
                       "disagree_rate": disagree, "flagged": flagged,
                       "truly_flipped": np.isin(np.arange(len(y)), flip)})

top = report.sort_values("margin").head(len(flip))
print("recall of injected errors in the top-N by margin:",
      round(top["truly_flipped"].mean(), 3))
print("flagged by per-class threshold:", int(flagged.sum()),
      "of which truly flipped:", int(report.loc[flagged, "truly_flipped"].sum()))

The numbers this prints depend entirely on class_sep — on how separable the classes are. That dependence is the finding, not a caveat: on a well-separated problem the flipped rows stand out sharply, and on an overlapping one the flagged set fills up with genuinely ambiguous rows. Run it on your own data with a small deliberately corrupted holdout and you get a precision estimate for your specific problem, which is the only number worth acting on.

What to do with the flagged rows

Do not auto-delete. This is the important instruction. The detector cannot distinguish a wrong label from a hard example, and hard examples near the decision boundary are exactly the rows that carry the most information about where the boundary is. Deleting every row the model finds surprising trains the model to agree with itself and produces a cleaner-looking score on a dataset that no longer contains the difficult cases.

1. Sort by margin and review the top 100 by hand, against the source record rather than against the label. This is a couple of hours and it tells you the base rate of real errors in the flagged set.
2. Look for pattern in what you find: one annotator, one date range, one upstream source, one class direction. A patterned error is fixable at the source, and fixing it is worth more than any relabelling.
3. Relabel where the correct answer is clear. Where it is genuinely ambiguous, that ambiguity is a property of the problem — consider recording it, rather than forcing a label.
4. Clean the evaluation set first. It is smaller, and every downstream decision depends on it. A model trained on slightly noisy labels and evaluated on clean ones is a much better position than the reverse.
5. Re-run the detection after cleaning. Removing one systematic error source frequently reveals a second one that was masked by it.

One last check before any of this. A row that every fold contradicts can also indicate that a feature is inconsistent rather than the label — a unit that changed midway through the history, a field whose meaning was redefined. If the flagged rows cluster in time, look at the features for that window before you touch the labels, and confirm that nothing in the feature set is quietly encoding the outcome, which is the subject of target leakage detection.