ROC and Precision-Recall Curves: Reading Them Right

A model does not have a precision. It has a score for every example and a threshold you chose, and the curves are what you get when you refuse to choose the threshold yet. Which curve you should be looking at is settled by one number: how rare the positive class is.

What a curve actually is

Sort every example by the model’s score. Sweep a threshold from the top of that list to the bottom, and at each position compute two numbers from the resulting confusion matrix. Plot one against the other and you have a curve.

• ROC plots true positive rate, TPR = TP/(TP+FN), against false positive rate, FPR = FP/(FP+TN).
• Precision-recall plots precision, TP/(TP+FP), against recall, which is the same quantity as TPR.

Both describe the same sorted list. They differ only in what they divide by, and that is the whole of the argument about which to use.

Two properties follow from that construction and are worth having in mind. First, the curve is a step function with one step per example, not a smooth arc: on 200 test rows it has at most 200 corners, and the elegant curve your plotting library draws is interpolation. Second, both curves are invariant to any monotone transform of the scores, because only the ordering is used — so a model whose outputs are uncalibrated probabilities, or arbitrary unbounded logits, produces exactly the same curves as its calibrated twin. That invariance is convenient when comparing models with incomparable score scales, and it is the reason ranking quality and calibration have to be checked separately.

Why ROC ignores prevalence

Look at the denominators. TPR divides by the number of actual positives; FPR divides by the number of actual negatives. Each is computed within one class. So if you multiply the number of negatives by ten — same classifier, same score distribution, just more negatives — FP scales up and TN scales up, and their ratio, FPR, is unchanged in expectation. The ROC curve does not move.

Precision does not have this property, because its denominator TP + FP mixes the two classes. Ten times as many negatives means roughly ten times as many false positives sitting in the same denominator as an unchanged TP. Precision collapses.

Write it exactly. With prevalence π (the fraction of examples that are positive), the counts at a given threshold are proportional to π·TPR for true positives and (1−π)·FPR for false positives, so:

                  π · TPR
precision = ─────────────────────────
             π · TPR + (1 − π) · FPR

TPR and FPR are properties of the classifier. π is a property of the world. Precision is a function of both — which is why it is the number that changes when you deploy a model trained on a balanced sample into traffic that is not balanced, without anything about the model changing at all.

The same classifier, two prevalences

Take a decent operating point: TPR = 0.90, FPR = 0.10. Substitute two prevalences into the formula above.

balanced test set, π = 0.50
  precision = (0.50 × 0.90) / (0.50 × 0.90 + 0.50 × 0.10)
            = 0.450 / (0.450 + 0.050) = 0.900     →  90%

production traffic, π = 0.01
  precision = (0.01 × 0.90) / (0.01 × 0.90 + 0.99 × 0.10)
            = 0.009 / (0.009 + 0.099) = 0.083     →  8.3%

Identical model. Identical ROC curve. Identical AUC. Precision falls from 90% to 8%, meaning eleven false alarms for every real catch. This is the single most common way a model that looked excellent in a notebook becomes an alerting system nobody reads, and no amount of staring at the ROC curve would have predicted it — because the ROC curve is, by construction, the part that did not change.

The published version of this argument is worth citing when someone insists on AUC: Saito and Rehmsmeier (PLOS ONE, 2015) argue the precision-recall plot is more informative than ROC on imbalanced data, and Davis and Goadrich (ICML, 2006) showed the relationship between the two spaces — a curve that dominates in ROC space dominates in PR space and vice versa, while the visual impression of quality differs enormously.

What AUC means, and its baseline

ROC-AUC has an exact interpretation that is more useful than “area under the curve”: it is the probability that a randomly chosen positive is scored above a randomly chosen negative. That makes 0.5 the coin-flip baseline and 1.0 perfect ranking, and it makes AUC a statement about ordering only — it says nothing about whether the scores are calibrated probabilities.

PR-AUC has no such fixed baseline. A random classifier’s PR-AUC is approximately π, the prevalence. So a PR-AUC of 0.40 is spectacular at π = 0.01 (forty times baseline) and poor at π = 0.50. Any PR-AUC quoted without the prevalence beside it is uninterpretable, and this is the most common omission in model reports.

Both areas share a further limitation: they are statements about ranking, not about probabilities. A model can have an AUC of 0.95 and be badly calibrated — every score squashed into the range 0.4 to 0.6, say — because applying any monotone transform to the scores leaves the ordering, and therefore both curves, completely unchanged. If a downstream system multiplies your probability by a cost, or shows it to a person, ranking quality is not the property you need. Check calibration separately with a reliability diagram or a proper scoring rule such as the Brier score, and remember that any resampling you did for class imbalance has already broken calibration by construction.

Choosing the point you will ship

The curves exist to help you pick one threshold; they are a stage in the argument, not the conclusion. Three habits make that choice honest:

• Evaluate at the prevalence you will actually see. If you rebalanced the training set, do not evaluate on the rebalanced one. If your test set was constructed by sampling positives, correct the precision back using the formula above rather than reporting the sampled number.
• Pick the point by cost, not by curve shape. “The elbow” is not a decision rule. The false-negative to false-positive cost ratio is, and the expected-cost calculation takes a minute.
• Report the operating point, not just the area. AUC summarises thresholds you will never use, including absurd ones. The number that predicts your experience is precision and recall at the threshold you shipped, on traffic with the prevalence you have.