Labelling Data Without a Labelling Team

You need a few thousand labelled examples — for an evaluation set, a classifier, a fine-tune — and there is no labelling team. The pattern that works is not “have a model do it”; it is having a model do it twice and having a person settle the arguments.

The shape that works

Four steps, and the order matters more than any individual choice inside them.

• Write the guidelines and label 50 items by hand. Before any automation. This is where you discover that your label set is ambiguous, and it is much cheaper to discover that at 50 items than at 5,000.
• Pre-label everything with a model, using the guidelines as the prompt and asking for a label plus a one-sentence justification. The justification is not decoration — it is what makes adjudication fast for the human.
• Label again with a second, independent pass — a different model, or the same model with a different prompt framing — and keep both answers.
• A human adjudicates only the disagreements, plus a random sample of the agreements to estimate the error rate on the part nobody looked at.

The second pass is the part people skip, and it is the part that makes the whole thing work — for reasons that are about routing rather than about accuracy.

Routing on disagreement, not confidence

The obvious alternative is to route on the model’s own confidence: send the low-confidence items to a human. It works less well than it sounds, because a model’s stated confidence is not well calibrated and it is confidently wrong on exactly the items you most need caught.

Disagreement between two independent passes is a much better signal. It requires no calibration, it is cheap to compute, and it directly identifies items where the guidelines are ambiguous — which is information you want anyway, because an item two competent labellers disagree about is usually an item where the label scheme is wrong rather than an item that is hard.

def route(items, label_a, label_b):
    """label_a/label_b: independent labelling functions."""
    auto, review = [], []
    for it in items:
        a, b = label_a(it), label_b(it)
        if a.label == b.label:
            auto.append((it, a.label, "agreed"))
        else:
            review.append((it, a, b))          # a human sees both + reasons
    return auto, review

# Then: sample the agreed pile too, or you have no idea what is in it.
audit = random.sample(auto, k=min(200, len(auto)))

The audit line is not optional. Without it you know the error rate on the reviewed items (zero, by construction) and nothing at all about the 80% nobody looked at, which is the only part of the dataset that matters for the claim “these labels are good”.

The arithmetic

Every number below is an assumption. The conclusion is not sensitive to which ones you change, which is the point.

Assume N = 5,000 items. Assume a human takes t = 60 seconds per item labelling from scratch, and t_adj = 20 seconds adjudicating a disagreement where two candidate labels and their justifications are already on screen. Assume the two model passes agree on a fraction a of items.

all human:      N * t                       = 5000 * 60s   = 83.3 h

pre-label:      2 * N * (tokens * price)    ... the model cost
adjudicate:     N * (1 - a) * t_adj         disagreements
audit:          n_audit * t                 sample of the agreed pile

with a = 0.85, n_audit = 200:
  adjudicate = 5000 * 0.15 * 20s = 4.2 h
  audit      =  200 * 60s        = 3.3 h
  human total                    = 7.5 h        (vs 83.3 h)

An 11× reduction in human time, and the model cost is two calls per item over 5,000 items — for a short classification prompt that is a rounding error against eleven person-days. Vary a and the conclusion holds: even at a = 0.6, human time is 19 hours.

The number that would break it is t_adj approaching t — if adjudicating is as slow as labelling from scratch, the routing buys nothing. That is why the justifications matter, and why the adjudication interface should show both labels and both reasons on one screen with two keys to choose between them.

One warning about where these labels may go. If the dataset will be used to train or fine-tune a model, labels produced by another model inherit that model’s biases and its blind spots, and the adjudicated disagreements are the only part a human has actually verified. That is usually fine for an evaluation set built to compare configurations, and it is a real consideration for a training set meant to teach a capability. It is also a licensing question worth checking before you start, since some providers restrict using their outputs to train competing models.

The guidelines are the real artefact

The labels are an output. The guidelines document is the thing you will still be using in a year, and it should be treated as code: versioned, reviewed, and changed only with a note about which items need relabelling.

A useful guideline has a definition per label, at least two positive examples and — most importantly — the near-miss cases with an explanation of which way they go and why. The near-misses are what disagreement analysis produces, so the loop is: label, find disagreements, resolve them, write the resolution into the guidelines, bump the version. After two rounds of that the agreement rate rises, which lowers the human cost in the arithmetic above — the guidelines are what pays for themselves.

Put the guidelines in the pre-labelling prompt verbatim. If a model cannot follow a written definition, a new labeller reading the same document will struggle too, and that is a signal about the definition.

Measuring whether the labels are any good

Raw agreement percentage overstates quality when the classes are unbalanced: two labellers who always answer “no” on a dataset that is 95% “no” agree 95% of the time and have demonstrated nothing. Cohen’s kappa corrects for agreement expected by chance:

from sklearn.metrics import cohen_kappa_score

k = cohen_kappa_score(labels_a, labels_b)
# kappa = (p_observed - p_chance) / (1 - p_chance)
# 1.0 identical; 0.0 no better than chance; negative = systematic disagreement

Compute it between the two model passes, and between a human and each model on the hand-labelled 50. A low human-model kappa with a high model-model kappa is the diagnostic worth knowing: the two models agree with each other and disagree with you, which usually means the guidelines say something different from what you meant.

Finally, version the dataset like any other artefact. A label set that changed between two evaluation runs makes the two numbers incomparable — the manifest should record the label version alongside everything else.