Judge Bias: Length, Position and Self-Preference

Three biases of model judges are documented well enough that you should assume yours has them until you have checked. The effect sizes in the literature come from other people’s judges, other people’s prompts and other people’s tasks — so the useful thing to publish is not a number, it is the test.

Position bias

When two candidate answers are presented for comparison, a judge’s verdict depends partly on which one came first. Zheng et al., 2023 documented this for the MT-Bench judge setup and proposed the standard mitigation — running every comparison in both orders and only counting a result when the two agree. Wang et al., 2023, “Large Language Models are not Fair Evaluators”, examined the same effect directly and proposed calibration and multiple-evidence strategies against it.

The mechanism is worth understanding because it tells you when to expect it to be worst. The judge reads a sequence and produces a verdict token; the two candidates occupy different positions in that sequence and are therefore not interchangeable inputs. Bias is strongest exactly where you care most: on close pairs. On an obvious blowout, position does not move the verdict; on a genuine near-tie, position can be most of the decision.

What to do with the disagreements is a real choice with consequences. Counting a flip as a tie is the conservative option and it is what most published setups do — it shrinks your measured win rate towards 0.5, which costs statistical power but never invents a preference. Discarding flipped pairs entirely is tempting and subtly wrong: the pairs that flip are systematically the close ones, so dropping them leaves you estimating the win rate on an easier subset and reporting a more confident number about a different question. If you discard, report how many you discarded, because that count is itself the most honest summary of how much your judge could resolve.

Verbosity bias

Judges tend to prefer longer answers, over and above the content in them. Zheng et al. describe it as a failure mode of model judges, and Saito et al., 2023 studied it as verbosity bias in preference labelling specifically. The practical significance is large because length is a knob: a system prompt saying “be thorough” can move a judge-scored metric without changing the quality of anything.

The response from the AlpacaEval maintainers is the clearest evidence that this is a first-order problem rather than a curiosity. Dubois et al., 2024 introduced a length-controlled version of AlpacaEval that statistically adjusts for output length precisely because the uncontrolled version was gameable by writing more, and the length-controlled variant became the headline number.

Note the trap in testing this. You cannot conclude bias from “longer answers score higher”, because longer answers are often better answers. The test has to hold content constant and vary only length, which is what the harness below does.

There is a cheaper diagnostic worth running against any judge-scored eval you already have: regress the per-item score on output length in tokens, pooled across candidates, and look at the slope. It cannot separate bias from genuine quality on its own, so it proves nothing — but a steep positive slope tells you that length explains a large share of the variance in your metric, and a metric whose strongest single predictor is character count is one to be suspicious of before shipping a decision on it.

Self-preference

A model used as a judge tends to rate its own outputs more favourably than a neutral party would. Zheng et al. call this self-enhancement bias. Panickssery, Bowman and Feng, 2024, “LLM Evaluators Recognize and Favor Their Own Generations”, went further and connected the preference to self-recognition — models can identify their own text, and the strength of that ability tracks the strength of the preference.

This is the bias with the cheapest fix and the one most often ignored, because the convenient judge is usually the strongest model you have access to, which is usually also a candidate.

It also arrives through routes that are easy to miss. A judge and a candidate from the same vendor and generation share training lineage even when they are different sizes. A candidate that was fine-tuned on outputs distilled from the judge’s family inherits its stylistic fingerprints. And a rubric written by iterating against one model’s output until it “worked” encodes that model’s habits into the criteria themselves, which is self-preference laundered through a document. The only reliable check is the cross-judged comparison: if a candidate’s advantage shrinks when a judge from an unrelated family scores it, you have found something regardless of which route produced it.

The harness

Three tests, one file, no dependencies beyond whatever function you already have for calling a model. Each returns a rate you can track over time and compare across judge models.

import random

# judge(prompt) -> "A" | "B" | "TIE"; supply your own caller.
# pairs: list of (question, answer_x, answer_y) drawn from YOUR eval set.

TEMPLATE = """Question:
{q}

Response A:
{a}

Response B:
{b}

Which response better satisfies the criteria below? Reply with exactly
one of: A, B, TIE.
{criteria}"""


def position_bias(judge, pairs, criteria):
    """Swap consistency. Flips are the bias; first_wins shows its direction."""
    agree = flips = first_pref = decided = 0
    for q, x, y in pairs:
        v1 = judge(TEMPLATE.format(q=q, a=x, b=y, criteria=criteria))
        v2 = judge(TEMPLATE.format(q=q, a=y, b=x, criteria=criteria))
        # v2 is in swapped coordinates: "A" there means y won.
        v2n = {"A": "B", "B": "A", "TIE": "TIE"}[v2]
        if v1 == v2n:
            agree += 1
        else:
            flips += 1
            if v1 != "TIE":
                first_pref += 1   # order 1 chose whatever was shown first
            if v2 != "TIE":
                decided += 1
    n = len(pairs)
    return {
        "swap_consistency": agree / n,
        "flip_rate": flips / n,
        "flips_favouring_position_1": first_pref / max(flips, 1),
    }


PADDING = ("Restate the answer above in different words, adding no new "
           "facts, no new recommendations and no new caveats. Keep every "
           "claim identical.")

def verbosity_bias(judge, gen, items, criteria):
    """Content held constant, length varied. Any preference is bias."""
    prefers_longer = decided = 0
    for q, base in items:
        longer = gen(f"{base}\n\n{PADDING}")
        if len(longer) < len(base) * 1.5:
            continue                      # padding did not take; skip item
        v = judge(TEMPLATE.format(q=q, a=base, b=longer, criteria=criteria))
        if v == "TIE":
            continue
        decided += 1
        prefers_longer += (v == "B")
    # 0.5 is unbiased. Run the swap too: this call has the long one second.
    return {"prefers_longer": prefers_longer / max(decided, 1), "n": decided}


def self_preference(judge_a, judge_b, outputs, criteria):
    """outputs: {model_name: {item_id: text}} including judge_a's own model.
    Compare judge_a's ranking with a judge from a different family."""
    def winrate(judge, own):
        wins = total = 0
        ids = list(next(iter(outputs.values())).keys())
        for i in ids:
            for other in outputs:
                if other == own:
                    continue
                x, y = outputs[own][i], outputs[other][i]
                if random.random() < 0.5:
                    v = judge(TEMPLATE.format(q=i, a=x, b=y, criteria=criteria))
                    wins += (v == "A")
                else:
                    v = judge(TEMPLATE.format(q=i, a=y, b=x, criteria=criteria))
                    wins += (v == "B")
                total += v != "TIE"
        return wins / max(total, 1)
    own = "the model judge_a is built on"
    return {"self_judged": winrate(judge_a, own),
            "cross_judged": winrate(judge_b, own)}

Two notes on using it honestly. The verbosity_bias test must itself be run in both orders and averaged, or you are measuring position bias again. And self_preference is only interpretable as a gap between the two win rates — a single win rate from a single judge tells you nothing, because the model might genuinely be better.

What actually reduces each one

None of these makes a judge unbiased. They make the bias bounded and visible, which is the achievable goal — the same goal you would have with any instrument that has a known systematic error.

Two further tendencies are worth watching for even though they are less formally characterised than the three above. Judges reward formatting: an answer with headings and bullet points often reads as more thorough than the same content as a paragraph, which means a prompt change that adds markdown can move a quality metric on its own. And judges tend towards agreeableness — an answer that confidently asserts something tends to be preferred over one that correctly says the question cannot be answered from the given context, which is precisely backwards for retrieval-augmented systems where declining to answer is the desired behaviour. Both are testable with the same content-controlled design as the verbosity test: hold the substance fixed, vary the one thing, see whether the verdict moves.