Detecting Label Noise in an Existing Dataset

Some fraction of the labels in your dataset are wrong. The useful question is not whether — it is which ones, and there is a method that finds them without any additional labelling.

How common label errors are

Northcutt, Athalye and Mueller audited the test sets of ten of the most widely used benchmark datasets across vision, language and audio, in Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (NeurIPS 2021 datasets and benchmarks track). They found label errors averaging at least 3.3% across the ten, with the ImageNet validation set alone containing thousands of erroneous labels — on the order of 6% of it.

The consequence they draw is the interesting one. It is not merely that scores are slightly wrong. On the more error-prone benchmarks, correcting the labels can change which model ranks highest — a larger model that fits the noisy labels more faithfully can lose to a smaller one once the errors are removed. If that is true of the most scrutinised public test sets in the field, assume it is true of the set your team assembled in three weeks.

It also means the ceiling on your metric is not 100%. A test set with 4% wrong labels caps measured accuracy near 96%, and any effort spent chasing the last few points is chasing the noise.

Confident learning, in plain terms

Confident learning (Northcutt, Jiang and Chuang, JAIR 2021; implemented in the cleanlab library) is the standard method, and the idea is simpler than the paper.

Train a model with cross-validation so that every example receives a predicted probability from a model that never saw it — this is the load-bearing detail, because in-sample predictions memorise the noisy label and reveal nothing. Then look for examples the model confidently assigns to a class other than their given label. Aggregate those disagreements into a matrix of counts: for each pair of classes, how many examples are labelled i but confidently look like j. That matrix estimates the joint distribution of noisy and true labels, and from it you get both an estimated overall noise rate and a ranked list of the individual rows most likely to be wrong.

Two things make it work in practice. It uses class-specific thresholds rather than one global confidence cutoff, which stops it from flagging everything in the classes a model is generally unsure about. And it produces a ranking, so a human reviewing the top two hundred rows is reviewing the two hundred most likely errors rather than a random sample.

The procedure

import numpy as np
from sklearn.model_selection import cross_val_predict
from cleanlab.filter import find_label_issues
from cleanlab.dataset import health_summary

# 1. OUT-OF-SAMPLE probabilities. Every row scored by a model that never
#    saw it. Using in-sample predictions here silently breaks the method.
probs = cross_val_predict(model, X, y, cv=5, method="predict_proba")

# 2. Rank the rows by how likely the given label is wrong.
issues = find_label_issues(
    labels=y,
    pred_probs=probs,
    return_indices_ranked_by="self_confidence",
)
print(f"{len(issues)} candidate label errors out of {len(y)} "
      f"({len(issues)/len(y):.1%})")

# 3. Which class pairs are being confused? This is the finding that changes
#    the TAXONOMY rather than individual rows.
print(health_summary(labels=y, pred_probs=probs))

# 4. Review the top of the ranking BY HAND. Do not auto-delete.
for i in issues[:200]:
    print(y[i], "->", np.argmax(probs[i]), texts[i][:120])

For a text dataset with no trained classifier to hand, embeddings plus logistic regression is an adequate model for this purpose — you are not building a production classifier, you are producing probabilities good enough to rank disagreements. A weak model finds fewer errors; it does not find wrong ones.

Step 3 is the one to read carefully. If the class-confusion summary says most of your candidate errors are one pair of classes in both directions, you do not have a labelling problem. You have a taxonomy problem: the two categories are not distinguishable from the input, and no amount of re-labelling will fix that. Merging them, or writing the rule that separates them, is the actual fix.

Three checks that need no machinery

Before any of the above, three passes that cost minutes and routinely find more than the sophisticated method does:

• Duplicate inputs with different labels. Group by an exact hash of the input. Any group with more than one distinct label is a guaranteed error — at least one of them is wrong, by construction. This finds real errors with zero false positives and it is a single query.
• Near-duplicate inputs with different labels. The same idea through MinHash. Weaker evidence than exact duplicates, since near-duplicates can legitimately differ, but a high-yield list to review.
• Annotator-level statistics. Per annotator: label distribution, mean time per item, and agreement with the majority on overlapped items. One annotator whose distribution differs sharply from everyone else’s, or whose speed is triple the median, is a finding about a block of rows rather than about individual ones. This is the check with the highest yield per minute in the entire list.

What to do with the rows you find

The default instinct — delete the flagged rows — is usually wrong, and in the worst case is actively harmful: the flagged rows are disproportionately the hard cases, and deleting them makes the dataset easier rather than cleaner.

• Relabel the test set, always. Errors in a test set corrupt every decision made with it and there are few enough rows to fix by hand. This is where the effort goes first.
• Relabel training rows if the budget allows. A corrected row is worth more than a deleted one. Send the ranked list to annotators as a review queue — it is the highest-value queue you can give them, since every item is pre-selected for being probably wrong.
• Down-weight rather than delete when it does not. Training with a reduced weight on suspect rows keeps their input distribution while reducing the damage of their labels.
• Fix the taxonomy when the confusion is systematic. If two classes account for most of the flags, the guideline is the problem. Rewrite it, then re-label the affected slice, then re-run.
• Never let the flagging model be the judge. If you delete every row a model disagrees with, you have trained the next model to agree with this one and quietly removed everything it found hard. That is a self-confirming dataset, and it is the same self-referential trap as training on your own output, arriving through the labels instead of the text.