Active Learning: Labelling Only What Matters

Active learning is the idea that a model should choose what gets labelled next. It has a long literature, a real mechanism, and a published caution that is much less widely cited than the technique itself.

The idea, and the only reason it works

You have a large pool of unlabelled examples and a budget for labelling a small fraction of them. Random sampling spends that budget in proportion to how common each kind of example is, which means most of it goes on cases the model already handles. Active learning instead trains a model on what has been labelled so far, uses it to rank the unlabelled pool by how informative each item would be, and sends the top of that ranking to the annotators.

The mechanism is entirely dependent on one condition being true: examples must differ substantially in how much they teach. In a pool where every example is equally informative, active learning cannot beat random and only adds machinery. In a pool that is 95% near-duplicate easy cases with a thin tail of hard ones — which describes most production traffic — the difference is large, and that is the regime worth building for.

Four acquisition strategies

In practice the combination that survives contact with reality is uncertainty plus diversity: rank by uncertainty, then select a batch that spreads across clusters. Pure uncertainty sampling in batches is the single most common way to build an active learning loop that underperforms random.

There is also a failure mode specific to uncertainty that no amount of diversity selection fixes. A model is maximally uncertain about examples that are genuinely ambiguous, about examples that are mislabelled, and about examples that are simply garbage — a truncated document, a form submitted empty, a row in the wrong language. All three sit at the top of the uncertainty ranking, so an unfiltered active learning loop sends annotators a queue substantially composed of items that cannot be labelled correctly by anyone. Filter the pool for basic validity before ranking it, and expect the annotators to report the rest.

The break-even against random sampling

Before building this, work out whether it can pay. Every number below is a labelled assumption; substitute your own.

• Assume you need a target quality level that random sampling reaches at 10,000 labels.
• Assume active learning reaches the same level at 6,000 labels — a 40% reduction, which is at the optimistic end of what the literature reports and should be treated as a hoped-for figure, not a promise.
• Assume a fully loaded labelling cost of $0.80 per item.
• Assume engineering time at $600 per day.

Saving: 4,000 labels × $0.80 = $3,200. Cost: building the loop — inference over the pool, a ranking step, batch selection, retraining, and the orchestration around all of it — is realistically three to five days of work, so $1,800 to $3,000, plus the inference cost of scoring the pool repeatedly, plus ongoing maintenance.

The conclusion is uncomfortable and useful: at a 10,000-label budget, active learning roughly breaks even in the best case and loses in the median one. The arithmetic only turns decisively positive when at least one of these holds — the labelling budget is large (six figures of items), the per-item cost is high (a clinician, a lawyer, an engineer rather than a crowd worker), or the loop is permanent and amortises across many labelling rounds rather than one.

Which is why the highest-value version of this idea is usually not a loop at all. One pass of uncertainty-ranked triage — score the pool once with whatever model you have, label the most uncertain few thousand, stop — captures much of the benefit for a day of work rather than a week. Build the full loop only after the one-pass version has proved the ranking is informative.

The loop

import numpy as np
from sklearn.cluster import KMeans

def margin_uncertainty(probs):
    """Smallest gap between the top two classes. Better than max-probability:
    it distinguishes 'unsure between two' from 'unsure among twelve'."""
    top2 = np.sort(probs, axis=1)[:, -2:]
    return -(top2[:, 1] - top2[:, 0])          # higher = more uncertain

def select_batch(pool_probs, pool_embeddings, k=200, oversample=5):
    """Uncertainty to shortlist, diversity to choose within it.
    Skipping the second step is how batch active learning fails."""
    scores = margin_uncertainty(pool_probs)
    shortlist = np.argsort(scores)[-k * oversample:]
    labels = KMeans(n_clusters=k, n_init=10, random_state=0) \
                .fit_predict(pool_embeddings[shortlist])
    picked = []
    for c in range(k):                          # most uncertain item per cluster
        members = shortlist[labels == c]
        if len(members):
            picked.append(members[np.argmax(scores[members])])
    return picked

def run(pool, labelled, rounds=8, k=200, holdout=None):
    """Note the two guards: a RANDOM control arm, and a fixed holdout set
    that is never chosen by the acquisition function."""
    control = []
    for r in range(rounds):
        model = train(labelled)
        probs, emb = model.predict_proba(pool.X), pool.embeddings
        picked = select_batch(probs, emb, k)
        labelled += annotate(pool.take(picked))
        control  += annotate(pool.random(k))     # the comparison you need
        print(r, evaluate(model, holdout))       # holdout is RANDOM, always
    return labelled, control

Two guards in that code do most of the work of keeping you honest. The random control arm is the only way to know the loop is beating random rather than merely producing a rising curve — a curve rises either way. And the holdout set must be randomly sampled and fixed, because evaluating on data the acquisition function selected measures the acquisition function, not the model.

One more failure mode worth naming: the cold start. Uncertainty from a model trained on fifty examples is close to noise, so the first two or three rounds should be random or diversity-based. Active selection begins to earn its keep only once the model is good enough for its uncertainty to mean something.

The published warning

Lowell, Lipton and Wallace published Practical Obstacles to Deploying Active Learning (2019), and the finding is the one to weigh before committing. A dataset acquired actively is acquired with respect to a particular model: the examples were chosen because that model found them uncertain. They report that such datasets transfer poorly — a successor model, or a different architecture, trained on the actively acquired set can perform worse than one trained on a randomly sampled set of the same size. They also report that the gains over random are inconsistent across tasks and models to begin with.

For an ordinary machine-learning project that was an inconvenience. In a setting where the underlying model is replaced every few months, it is a serious risk: the dataset you spent a quarter acquiring was optimised against a model you no longer use. Three mitigations, all cheap:

• Keep a random fraction. Acquire, say, 70% actively and 30% at random. The random portion is what keeps the dataset representative and transferable, and it doubles as the control arm.
• Use a committee rather than one model. Selecting on the disagreement of several models produces a set that is less tied to any one of them.
• Record why every item was selected. Round number, strategy, uncertainty score, model version. Without that, a future reader cannot tell a representative dataset from a biased one, and the bias is invisible in the rows themselves — which is exactly what a datasheet is for.