Human Preference Data: Collecting It Cheaply

Preference data is the input to every alignment method that ranks outputs rather than imitating them. The expensive part is not the annotation tool. It is that two reasonable people disagree about a third of the interesting cases, and everything about the collection design is really about that number.

Why pairwise, not a scale

The instinct is to ask annotators to rate each response from 1 to 5. It produces data that looks tidier and is considerably worse, for reasons that are well understood:

• Scales are not calibrated between people. One annotator’s 4 is another’s 3. Averaging across annotators averages across two different scales, and the variance you measure is partly a property of your workforce.
• Scales are not calibrated within a person over time. Ratings drift across a session as the annotator recalibrates against what they have recently seen. The first hour and the fourth hour are different instruments.
• Comparison is the easier judgement. “Which of these two is better” is a question people answer quickly and consistently; “how good is this on an absolute scale” is a question that requires an internal standard nobody has.
• The training methods want pairs anyway. Reward modelling and direct preference optimisation both consume chosen/rejected pairs, so a scale has to be converted into pairs eventually, losing information on the way. See DPO versus PPO for what each does with them.

The design that follows: show two responses to the same prompt, ask which is better, and allow a tie. Allowing ties matters — forcing a choice on genuinely equivalent responses manufactures a preference signal out of a coin flip, and that noise goes straight into the reward model. The pairwise-versus-scoring comparison covers the same trade-off from the evaluation side.

What agreement rate to expect

This is the number that surprises people, so it is worth setting expectations from the published record rather than from optimism. The InstructGPT paper (Ouyang et al., 2022) reports inter-annotator agreement rates for its labellers in the region of the low-to-mid seventies per cent — that is, trained annotators working to a written guideline agreed with each other on roughly three comparisons in four. Other RLHF datasets report figures in a broadly similar band.

Three consequences follow, and all three change how you design the collection:

• There is a ceiling on any model trained on this data. A reward model cannot be more consistent than the signal it was fitted to. If humans agree 73% of the time, a reward model matching human preference 75% of the time is close to the achievable limit, not a disappointing result.
• Volume does not fix disagreement. Where the disagreement is genuine — the two responses really are equally good, or the guideline does not say which criterion wins — more annotations average the noise but do not resolve it. What resolves it is a better guideline.
• Disagreement is your most valuable signal. A comparison two annotators disagree on is either an ambiguous case worth documenting or an underspecified rule worth fixing. Route them to a reviewer rather than discarding them.

Measuring your own agreement

Raw percentage agreement overstates the case, because two annotators picking at random on a binary choice agree half the time. Cohen’s kappa corrects for chance for two annotators; Krippendorff’s alpha handles more than two and missing data. Overlap a fraction of every batch — 10% is a common choice — so that agreement is measured continuously rather than once at the start.

from collections import Counter
from sklearn.metrics import cohen_kappa_score

def overlap_batch(items, annotators, overlap=0.10, seed=0):
    """Route 10% of items to two annotators instead of one. Cheap insurance:
    without it you have no idea whether your labels mean anything."""
    import random
    rng = random.Random(seed)
    assignments = []
    for it in items:
        a = rng.choice(annotators)
        assignments.append((it, a))
        if rng.random() < overlap:
            b = rng.choice([x for x in annotators if x != a])
            assignments.append((it, b))
    return assignments

def agreement(pairs_a, pairs_b):
    """pairs_* are labels ('A', 'B', 'tie') for the SAME items, same order."""
    raw = sum(x == y for x, y in zip(pairs_a, pairs_b)) / len(pairs_a)
    kappa = cohen_kappa_score(pairs_a, pairs_b)
    return dict(raw=raw, kappa=kappa)

def disagreement_report(items, labels_a, labels_b):
    """The output that is actually worth reading: WHERE they disagreed."""
    rows = [(it, a, b) for it, a, b in zip(items, labels_a, labels_b) if a != b]
    by_category = Counter(it["category"] for it, _, _ in rows)
    return rows, by_category.most_common()

Interpret kappa cautiously. It is sensitive to the marginal distribution, so a task where one option wins 90% of the time can show a low kappa alongside a high raw agreement, and neither number alone tells the story. Report both, and read the disagreements — the category breakdown is where the actionable finding is.

The interface decisions that matter

Small choices here move agreement more than almost anything else you can do downstream.

Where the cost actually goes

“Cheaply” usually gets interpreted as finding cheaper annotators, which is the lever that damages the data most. The costs that are genuinely reducible are elsewhere:

• Do not annotate pairs that are not close. If one response is obviously better, the comparison teaches the reward model almost nothing and costs the same as a hard one. Pre-filter with a cheap model and send humans the pairs where the models disagree or the scores are close — the active learning idea applied to preferences.
• Do not annotate what a verifier can decide. If one response fails to compile, fails the schema or gets the arithmetic wrong, the pair is resolved without a human. Run the verifiers first.
• Deduplicate the prompts. Preference data collected from production traffic contains the same request many times over. Cluster and sample rather than annotating the same question forty times.
• Spend the budget on the guideline. The highest-return hour in this whole process is the one spent turning three disagreements into a written rule. It raises agreement on every subsequent item, which is the only intervention here that compounds.
• Use model judges for coverage, humans for calibration. A model judge can label far more pairs than you can afford humans for. What it cannot do is tell you whether it is right — so keep a human-labelled sample, measure the judge against it, and treat that measured agreement as the error bar on everything the judge produced. Judge bias is the specific list of ways it goes wrong.