AI in Medical Diagnosis: The Evidence Ladder a Model Has to Climb

This page is about how evidence for diagnostic models is constructed and read. It is not medical advice, it is not a guide to any specific system, and nothing here should be used to make a decision about anyone’s care.

The ladder

Most public discussion of diagnostic AI cites results from the first rung and language from the fifth. The rungs are not opinions; they are the standard hierarchy any diagnostic technology climbs, and each one answers a question the one below it cannot.

Why the numbers move in a clinic

The most common misreading is treating sensitivity and specificity as if they told you what a positive result means. They do not, and the gap is arithmetic rather than a matter of study quality.

Take a model with 95% sensitivity and 90% specificity — figures used here purely as an illustration, not as anyone’s reported performance. In a referral population where 30% of those tested have the condition, a positive result is right about 80% of the time. Move the same model, unchanged, to a screening population where 1% have the condition:

prevalence            1%
true positives        0.01 * 0.95  = 0.0095
false positives       0.99 * 0.10  = 0.0990
positive predictive value
  = 0.0095 / (0.0095 + 0.0990)     = 8.8%

Nine out of ten positives are wrong, with no change to the model at all. Every one of those is a patient who gets a follow-up investigation, and the harms and costs of those investigations are part of the evaluation, not an externality. This is why a study population has to be described in detail before a performance figure means anything, and why moving a validated model from a specialist setting to a screening setting is a new evaluation rather than a deployment.

Two related points. The operating point matters more than the summary score: a single number that averages over all thresholds tells you nothing about the trade-off you will actually run at, and the choice of threshold is a clinical judgement about which error is worse. And the comparator is usually wrong — a model compared against one clinician reading in isolation is not being compared against practice, where the clinician has the history, prior imaging, other test results and the option to ask.

What the model may have learned instead

A well-documented case in medical imaging: a model trained on data pooled from several hospital systems learned features that identified which system an image came from — equipment differences, markers burned into the image, positioning conventions. Because disease prevalence differed between the systems, that information was predictive of the label. Performance on held-out data from the same hospitals looked excellent, and it degraded on a new hospital, because what had been learned was in part the hospital rather than the disease.

This is not an exotic failure; it is the medical instance of the proxy feature described in the leakage taxonomy, and it is why rung 2 is a genuine boundary rather than a formality. Related shortcuts recur constantly: an image annotation applied only after diagnosis, a scanner used preferentially for sicker patients, a chest drain visible in images of the condition it was placed to treat. Each is a real correlation in the data and none of them is the disease.

There is a second layer worth naming: the label. In most imaging datasets the ground truth is another clinician’s report, so the model is trained to reproduce human reads, agreement between humans bounds the achievable score, and systematic human errors are learned as targets. A study that establishes ground truth independently — by biopsy, by follow-up outcome, by adjudicated panel — is answering a harder and much more useful question.

What breaks on deployment

The most instructive published work in this area is not about accuracy at all. A prospective study of a deployed retinal screening system in real clinics found that image quality standards calibrated on curated research data caused a substantial share of real-world images to be rejected — clinic lighting, undilated pupils, and equipment that was not what the training data came from. Patients who would have received a same-visit result instead received nothing and were asked to return. The model’s accuracy on the images it did accept was not the problem.

• Automation bias. Clinicians shift towards the model’s answer, including when it is wrong, and the shift is larger for less experienced readers. A combination of clinician and model can perform worse than the clinician alone if the model is confidently wrong in a correlated way, which is why rung 4 measures the pair rather than the model.
• Workflow, not accuracy, decides adoption. Where the result appears, how long it takes, who is accountable for acting on it, and what happens when it is unavailable determine whether a correct output changes anything.
• Drift. Equipment is replaced, protocols change, case mix moves. A model that is not monitored after deployment degrades silently, and the monitoring plan is part of the intervention.
• Regulatory clearance is narrow. Authorisation attaches to a specific intended use, population and workflow — most commonly as an aid to a clinician rather than as an autonomous decision, with autonomous authorisations existing only for a small number of well-defined screening tasks. It is not a general statement that the model is good.

Reading a diagnostic claim

• Which rung, stated plainly? If external validation is absent, the claim is about a dataset.
• What was the population, and what was the prevalence? Without both, a predictive value cannot be computed and the headline figure is not interpretable.
• What was the ground truth? Another clinician’s read, or an independent outcome.
• What was the comparator, and was it realistic? Clinicians working as they normally do, with the information they normally have.
• Was the endpoint clinical? A change in a model metric is not a change in what happens to patients, and trials of AI interventions now have dedicated reporting guidelines precisely because that distinction kept being blurred.