Precision, Recall, F1 and When Accuracy Lies

Accuracy is the metric everyone reaches for and the one that fails first, and it fails in a specific, predictable place: whenever the thing you are looking for is rare. One worked example makes the whole family of metrics fall out.

One confusion matrix, worked

Ten thousand card transactions. One hundred are fraudulent — a 1% positive rate, which is generous by real-world standards.

Model A predicts “not fraud” for everything. It is correct on all 9,900 legitimate transactions and wrong on all 100 frauds, so its accuracy is 9,900 / 10,000 = 99.0%. It has never caught anything and never will. This is not a strawman: it is what a classifier trained with plain cross-entropy on this data converges toward if nothing is done about the imbalance.

Model B flags 400 transactions, of which 70 are genuinely fraudulent. Its confusion matrix:

                    actual fraud    actual legit
predicted fraud        TP =  70       FP =  330
predicted legit        FN =  30       TN = 9,570

accuracy  = (70 + 9,570) / 10,000        = 96.4%
precision = 70 / (70 + 330) = 70 / 400   = 17.5%
recall    = 70 / (70 +  30) = 70 / 100   = 70.0%
F1        = 2 × 0.175 × 0.70 / 0.875     = 28.0%

Model B is obviously the useful one and it scores worse on accuracy than the model that does nothing. That inversion is the entire reason the other metrics exist, and it is why “our classifier is 96% accurate” is a sentence that should prompt a question rather than a nod.

The denominators are the whole idea

Precision and recall have the same numerator. Everything that distinguishes them is underneath the line:

• Precision = TP / (TP + FP) — the denominator is what you flagged. It answers: of the things I alerted on, how many were real? This is the number the analyst working the queue experiences, and it is what determines whether they keep trusting the alerts.
• Recall = TP / (TP + FN) — the denominator is what existed. It answers: of the real cases, how many did I catch? This is the number the risk owner cares about, and it is the one that cannot be computed without labels for cases you never flagged.

That second observation has a sharp practical edge. Recall requires knowing about the frauds you missed, which typically means waiting for chargebacks, or paying to label a random sample. Teams that only measure precision — because it is easy, you just review your own alerts — are systematically blind to the failure that matters most.

The two also trade off by construction: lower the decision threshold and you flag more, so recall rises and precision falls. They are not independent qualities to be maximised, they are two ends of one dial.

F1, and why it is a harmonic mean

F1 is 2PR / (P + R), the harmonic mean of precision and recall. The harmonic mean is chosen because it is dominated by the smaller of the two. Compare a model with P = 0.9, R = 0.1:

arithmetic mean = (0.9 + 0.1) / 2                   = 0.500
harmonic mean   = 2 × 0.9 × 0.1 / (0.9 + 0.1)       = 0.180

An arithmetic mean would let a model hide a catastrophic recall behind a good precision. F1 does not. That is its one virtue, and it comes with two defects worth stating: it weights precision and recall equally, which is almost never what your problem does, and it ignores true negatives entirely, so it is not comparable across datasets with different prevalence.

Fβ fixes the first defect — β > 1 weights recall more, β < 1 weights precision more — and the honest way to set β is from costs, which is the next section. The macro/micro distinction matters too: macro-F1 averages the per-class F1 scores and therefore gives a rare class the same weight as a common one, while micro-F1 pools the counts and lets the common class dominate. Say which you used.

None of this makes accuracy a bad metric — it makes it a metric with a precondition. When the classes are roughly balanced and the two errors cost about the same, accuracy is the right summary and everything above is overhead. Two alternatives are worth knowing for when they are not: balanced accuracy, the mean of per-class recalls, which strips out prevalence; and the Matthews correlation coefficient, which uses all four cells of the confusion matrix and therefore cannot be inflated by a large true-negative count the way accuracy can. Whatever you pick, report the confusion matrix beside it. Every metric on this page is a lossy summary of those four numbers, and different readers want different summaries.

The metric nobody writes down

Every classification decision has a cost matrix, and most teams have never written theirs on paper. Do it, and the threshold stops being a matter of taste.

Suppose a missed fraud costs €200 and a false alarm costs €5 in review time and customer friction. Then for a threshold t producing FN(t) misses and FP(t) false alarms, expected cost is:

cost(t) = 200 · FN(t) + 5 · FP(t)

Model B above:  200 × 30 + 5 × 330 = 6,000 + 1,650 = €7,650
Model A (flags nothing): 200 × 100 + 5 × 0        = €20,000

The break-even is where one extra catch is worth the false alarms it brings: at these costs you should accept up to 200 / 5 = 40 extra false alarms to catch one more fraud. That single ratio tells you which end of the precision-recall dial to sit at, and it is a business input, not a modelling one. Sweep the threshold, compute the cost at each, pick the minimum, and report both the threshold and the assumed costs so the choice can be argued with.

The same trap in LLM evaluation

Every guardrail, router and content filter in a language-model system is a classifier on an imbalanced class, and the same arithmetic applies unchanged. A jailbreak detector where 0.5% of traffic is hostile can report 99.5% accuracy while catching nothing. A judge model that “agrees with humans 90% of the time” on a task where 90% of responses are fine has demonstrated nothing at all.

Ask for the confusion matrix, not the headline. And before believing a difference between two judges or two prompts, check whether the sample size can support it — with 200 eval items, a 3-point difference is inside the noise.