How to read a medical AI study without being fooled
A high AUC does not mean a better-off patient. Five questions that separate a promising result from one that can change practice.
5 March 2025 · 9 min
Key points
- The central question is not how good the model is, but what you compare it against.
- External validation and complete reporting matter more than one more point of performance.
- A retrospective, single-centre study does not justify a change in practice — however good the numbers look.
What is the comparator?
The first page of any study about a diagnostic model should answer one question: who is it competing with? An algorithm compared with itself, with an arbitrary threshold, or with a tired resident tells us very little about usefulness. The right comparator is usual practice in the centre where the model would be deployed — including its time pressure, fatigue and consequences.
Reader studies — where radiologists read the same cases with and without the algorithm — are more informative, but carry their own bias: readers know they are being evaluated, and cases are selected rather than consecutive.
Population and disease spectrum
A model can perform beautifully on a sample where disease is obvious and common, and fail in real triage where most patients are well. This is spectrum bias and it is the most common source of unjustified enthusiasm.
Read the exclusion criteria. Who was removed — patients with implants, with motion artefact, with poor-quality images — is exactly the population the model will meet on a Monday morning.
What the metrics hide
AUROC is a threshold-free measure and therefore convenient for authors. In practice the model runs at a chosen threshold, where sensitivity, specificity and above all positive predictive value in the real prevalence matter. A PPV of 40% in a service with 2% prevalence means six out of ten alerts are false — and alert fatigue within a fortnight.
Calibration is equally important and far less often reported: if the model says “30% risk”, how often is it actually 30%? Without calibration, the probabilities shown to a clinician are decoration.
External validation and time
The order of trust is simple: internal validation (same dataset, split) < external validation (another hospital, another scanner, another year) < prospective study < study measuring a clinical outcome. Each step costs money and each step is routinely skipped in press releases.
Change over time is underestimated: protocols shift, scanners are replaced, populations change. A model validated in 2021 on 2018 data needs monitoring, not just an approval.
What you do on Monday morning
Ask for complete reporting — TRIPOD+AI for prediction studies, PROBAST to appraise risk of bias — and check whether performance holds in the subgroups you treat: older adults, multimorbid patients, different ethnic populations.
Then ask who is accountable when the model is wrong. A system with no clear answer to that question is not ready for the bedside.