Weak Supervision and Programmatic Labelling

Weak supervision replaces the act of labelling ten thousand rows with the act of writing twenty rules that are each individually unreliable — and then does something non-obvious to work out how much to trust each rule without ever seeing a ground-truth label.

The shift: label the rule, not the row

Ratner and colleagues introduced the framing as data programming (NeurIPS 2016), and it became the Snorkel system (VLDB 2017 onward). The observation behind it is that domain experts are slow at labelling examples and fast at articulating heuristics. Ask a support lead to label three thousand tickets and it takes a week. Ask what makes a ticket a billing complaint and you get eight rules in ten minutes.

Each rule becomes a labelling function: a small program that looks at an example and either emits a label or abstains. The functions are allowed to be wrong, allowed to be correlated, allowed to cover only a slice of the data, and allowed to contradict each other. What the system does with that mess is the interesting part.

The payoff is not only speed. A labelling function is code — it lives in version control, it is reviewable, it can be re-run over a new corpus in seconds, and when the taxonomy changes you edit a rule instead of re-labelling a dataset. That maintainability is often worth more than the initial time saved.

What a labelling function looks like

Deliberately varied in kind: the mix is what makes the reconciliation step work, because functions that fail differently are what let the label model see anything at all.

ABSTAIN, BILLING, TECHNICAL = -1, 0, 1

def lf_keyword_billing(x):
    kws = ("invoice", "charged", "refund", "receipt", "vat", "double-billed")
    return BILLING if any(k in x.text.lower() for k in kws) else ABSTAIN

def lf_keyword_technical(x):
    kws = ("error", "500", "timeout", "crash", "stack trace", "not loading")
    return TECHNICAL if any(k in x.text.lower() for k in kws) else ABSTAIN

def lf_regex_amount(x):
    import re
    return BILLING if re.search(r"[€$£]\s?\d", x.text) else ABSTAIN

def lf_route_metadata(x):
    """External signal: which form the ticket came through."""
    return {"billing_form": BILLING, "bug_form": TECHNICAL}.get(x.source, ABSTAIN)

def lf_prior_ticket(x):
    """Weak transitive signal from an existing labelled history."""
    prev = history.last_label(x.customer_id)
    return prev if prev is not None else ABSTAIN

def lf_model_zero_shot(x):
    """A language model IS a labelling function — a noisy one with wide
    coverage. It belongs in the set, not above it."""
    out = classify(x.text, labels=["billing", "technical", "other"])
    return {"billing": BILLING, "technical": TECHNICAL}.get(out, ABSTAIN)

LFS = [lf_keyword_billing, lf_keyword_technical, lf_regex_amount,
       lf_route_metadata, lf_prior_ticket, lf_model_zero_shot]

Note the last one. Once zero-shot classification is cheap, a model prompt is simply another labelling function — high coverage, moderate accuracy, correlated with nothing else in the set. Treating it as one voice among several rather than as the answer is what keeps its systematic errors from becoming the dataset’s systematic errors.

The label model, which is the actual idea

Six functions vote on each example and disagree. Majority vote is the obvious reconciliation and it is wrong in a specific way: it weights a function that is right 95% of the time the same as one that is right 55% of the time, and it double-counts three functions that are really the same rule written three ways.

The label model does better, and the surprising part is that it needs no labelled data to do it. It estimates each function’s accuracy from the structure of agreement across the unlabelled corpus. The intuition: a function that agrees with the consensus wherever it votes is probably accurate; a function that fires often and agrees rarely is probably noise; and if you also model which functions are correlated, you can stop counting the same evidence twice. Fitting that — a latent-variable model over the observed vote matrix — yields per-function accuracies and, from them, a probabilistic label for each example.

Then comes the step that makes the whole thing worth doing: you train your actual classifier on those probabilistic labels. The end model generalises beyond what the rules literally match — it learns from the features, so it labels examples where every function abstained. That generalisation is the output. The labelling functions are scaffolding.

from snorkel.labeling import PandasLFApplier, LFAnalysis
from snorkel.labeling.model import LabelModel

L_train = PandasLFApplier(lfs=LFS).apply(df_train)     # n_examples x n_lfs

# Read this table BEFORE fitting anything. It is the diagnostic that matters.
print(LFAnalysis(L=L_train, lfs=LFS).lf_summary())
#   Polarity | Coverage | Overlaps | Conflicts   (+ Emp. Acc. if you have
#                                                 a small labelled dev set)

label_model = LabelModel(cardinality=2, verbose=True)
label_model.fit(L_train, n_epochs=500, seed=0)
probs = label_model.predict_proba(L_train)             # soft labels

# Train the end model on soft labels; drop rows where everything abstained.
mask  = (L_train != ABSTAIN).any(axis=1)
clf   = train_classifier(df_train[mask], probs[mask])

The summary table is the part to sit with. Coverage is the fraction of examples a function votes on, Overlaps how often it votes alongside another, and Conflicts how often it disagrees. A function with 2% coverage is not worth its maintenance; one with 90% coverage and high conflict is probably wrong; two functions with near identical coverage and no conflicts are one function.

The workflow that makes it converge

Weak supervision is iterative, and the iteration is the method rather than an implementation detail.

• Label a small dev set by hand first. Two hundred examples. Not to train on — to measure labelling functions against. Without it you are flying blind, and it is the cheapest two hundred labels you will ever buy.
• Write functions until coverage stops rising. Track what fraction of the corpus has at least one non-abstaining vote. When a new function adds nothing to coverage and nothing to accuracy, stop writing functions.
• Read what nothing covers. The uncovered slice is where the taxonomy is wrong or an entire category was forgotten. It is the most informative sample in the whole corpus and it costs nothing to produce.
• Prefer precision over coverage per function. A rule that fires on 3% of examples and is nearly always right is worth more than one that fires on 60% and is right two-thirds of the time. The label model can combine narrow accurate signals; it cannot rescue a broad wrong one.
• Watch for correlated functions. Three keyword rules over overlapping vocabularies are one piece of evidence wearing three hats. Either model the dependency or merge them.

Where it stops working

The honest boundary conditions, because this technique is oversold as often as it is undersold:

• When no heuristic exists. If a domain expert cannot articulate any rule — if the judgement is genuinely holistic — there is nothing to program. Sentiment on subtle text and quality judgement are the usual examples.
• When the classes are very rare. With a positive rate below a fraction of a per cent, agreement statistics on the positive class are estimated from almost nothing, and the label model has little to work with.
• When every function shares one blind spot. The method assumes errors are at least partly independent. If all your rules and your zero-shot classifier fail on the same subpopulation — a second language, a document format, a customer segment — the label model sees unanimous agreement and reports high confidence. This is the failure mode to actively hunt for, because it is invisible from the inside.
• When the labels need to be exactly right. Weak supervision produces a training set with known noise. That is fine for training a model and unacceptable for an evaluation set, a compliance record or anything a person will be held to. Build the eval set by hand regardless.

Where it fits alongside the neighbours: weak supervision is the right tool when you have a large unlabelled pool and articulable rules; active learning when you have a pool and a small budget of expert attention; and label-noise detection when you already have labels and suspect some of them. They compose well — weak supervision to get a first pass over everything, then noise detection over its output, then human attention only where the two disagree.