Bias in Language Models: Measuring It Properly

A model is reported to be biased and the number comes from a benchmark whose own authors’ critics have shown does not measure what its name claims. This page is about measuring the thing properly, which starts with deciding which thing you mean.

Four different claims called bias

• Representational harm. The model associates groups with stereotyped attributes, produces demeaning content, or erases a group. The harm is in the representation itself, independent of any decision.
• Allocative harm. A system using the model distributes something — an interview, a loan, a triage priority — unequally across groups in a way that is not justified. This is the one law mostly cares about.
• Performance disparity. The model is simply worse for some inputs: a dialect, a language, a name distribution, an accent. Not stereotype at all, and often the largest real-world effect.
• Viewpoint slant. The model’s outputs on contested political and moral questions lean one way. Measurable in some sense; but what the correct distribution of outputs would be is a value question with no neutral answer, and studies here are unusually sensitive to how the questions were written.

These have different measurements and different remedies. A model can show strong stereotype associations in an embedding probe and produce no allocative disparity in your pipeline, or the reverse. Reporting one as if it were the other is the most common error in this literature and in the coverage of it.

The measurement families

Association probes. The oldest family, from static word embeddings: measure whether group terms sit closer to some attribute terms than others. WEAT is the canonical instrument. Cheap, and only loosely connected to behaviour of a generative system.

Minimal-pair benchmarks. Present the model with two sentences differing only in a group term and compare likelihoods or choices. The coreference sets — Winogender and WinoBias — are the cleanest of these because the correct answer is determined by grammar, so a systematic error is unambiguous. StereoSet and CrowS-Pairs extend the idea to stereotype content.

Question answering under ambiguity. BBQ and similar sets ask a question in a context that does not determine the answer, then in one that does. The design is good: it separates whether the model falls back on a stereotype when information is missing from whether it ignores information when it is present.

Downstream audits. Run your actual pipeline on paired inputs that differ only in a group signal — names, pronouns, an institution — and compare the decisions. This is the family that measures allocative harm, and it is the only one whose result transfers to your deployment.

Documented weaknesses

• Construct validity. Blodgett and colleagues examined the widely used stereotype benchmarks and documented substantial problems with the items themselves — pairs that are not minimal, items where the stereotyped and non-stereotyped options are not comparable, and unclear or contested stereotype attributions. A score on an instrument with those defects does not cleanly mean what the instrument’s name says.
• Prompt sensitivity. Results on these benchmarks move with the template, the option ordering, the system prompt and the decoding settings. A comparison between two models measured with different harnesses is not a comparison.
• Contamination. Public benchmarks end up in training data. A model can improve on the metric without any change in the behaviour the metric was built to detect.
• Aggregation. A single bias score sums over groups and directions, so opposite errors cancel and a model that is badly miscalibrated for two groups in opposite directions can score well.
• Proxy conflation. Names are the usual proxy for group membership, and a name signals several attributes at once — ethnicity, national origin, class, generation, sometimes gender. An effect measured on names cannot be attributed to one of them.
• Model versus system. Nearly all published bias numbers are for a bare model. What you deploy has a system prompt, a retrieval layer, a filter, a temperature and a threshold, any of which can amplify or suppress the effect.

Why “unbiased” is not a well-defined target

This is the part most discussions omit, and it is a settled formal result rather than an opinion. Work by Kleinberg, Mullainathan and Raghavan, and independently by Chouldechova, established that standard fairness criteria are mutually incompatible: except in degenerate cases, a classifier cannot simultaneously be calibrated within groups and equalise false positive and false negative rates across groups when the base rates differ between those groups. Not “is hard to” — cannot, as a matter of arithmetic.

The consequence is that “make it fair” is not an implementable instruction. Someone must choose which criterion to satisfy, and that choice is a normative one about which error is worse to inflict on whom. It can be informed by evidence about the consequences of each error, and it cannot be settled by evidence. Systems that claim to be simply unbiased have made the choice silently.

What a defensible measurement looks like

• State the harm: representational, allocative, performance, or slant. One at a time.
• Measure the deployed pipeline, not the bare model, and record the full configuration.
• Use your own input distribution. A benchmark drawn from somewhere else measures a population you do not serve.
• Report disaggregated, with uncertainty. Group sample sizes are usually small enough that the interval matters more than the point.
• Say which fairness criterion you chose and why, given the impossibility result.
• Re-run on every model or prompt change, because none of this transfers across versions.

None of that is expensive relative to the cost of discovering the problem through a complaint. The reason it is rare is that it produces a document describing a disparity, and organisations are reluctant to create one — which is the same disclosure incentive that shapes model cards, and worth naming as the real obstacle rather than pretending the methods are the hard part.