Ideas · September 2026

Tested on the wrong question

A clinical AI model can score well on every metric its builders chose and still make care worse. That is not usually a failure of machine learning. It is a failure to agree, in advance, on what we were trying to find out.

When I say most clinical AI is tested on the wrong question, I mean three separate things. They get discussed as one problem, which is unhelpful, because they have different causes and different fixes.

The wrong target

The clearest case I know is an algorithm used across US health systems to flag patients for extra care. It worked. It predicted its target accurately. The target was annual health costs, used as a stand-in for health need.

Because less money is spent on Black patients at the same level of illness, the model concluded those patients were healthier. It cut the number of Black patients identified for extra care by roughly half. Changing the target — predicting illness rather than spending — raised the share of Black patients flagged from 17.7% to 46.5%.1

Nothing about the model was broken. It answered its question well. The question was the wrong one, and no amount of additional accuracy would have surfaced that.

The wrong data

The second failure is more familiar: models evaluated on retrospective data from a single site, then deployed somewhere else.

Of more than 500 AI-enabled devices authorised by the FDA, roughly 43% had no publicly available clinical validation data, and only about 28% had been tested prospectively.2 An earlier analysis of FDA approvals found evaluation frequently resting on single-site studies and curated research datasets, with limited reporting by sex or age.3

What that looks like in a hospital: a proprietary sepsis prediction model, live at hundreds of US sites, was externally validated at Michigan Medicine and scored an AUC of 0.63 — against the 0.76 to 0.83 its vendor had cited. Sensitivity was 33%. Positive predictive value was 12%.4

A positive predictive value of 12% means that for every hundred alerts a nurse works through, about 88 are wrong. That is not a statistical footnote. It is a shift in what every subsequent alert means to the person receiving it, and it compounds every hour of every shift.

The wrong place

The third failure is the one that surprises teams most, because the model is genuinely good.

A deep learning system for diabetic retinopathy screening, performing at specialist level in evaluation, was studied across eleven clinics in Thailand. Images taken in rooms that were not dark enough failed the system’s quality threshold and were rejected. Nurses reorganised their workflow around it. Internet speed determined whether results arrived while the patient was still there.5

The model never changed. The clinic was doing the testing, and it was asking a question the evaluation had not: does this work here, on these patients, with this staff, in this room?

The field already agrees

This is not a contrarian position. It is close to consensus among the people who study evaluation — they have simply written it in a form most teams never read.

DECIDE-AI exists specifically to cover the early, small-scale clinical evaluation that happens after an algorithm is built and before it is trial-ready — a stage that was going largely unreported.6 It sits alongside CONSORT-AI for trials and TRIPOD+AI for prediction models. A discipline does not produce three reporting guidelines for a problem it does not have.

What I would ask instead

Accuracy is necessary and it is not the finding. Before a pilot, these are the questions I want answered, and most of them are not statistical:

What decision changes? If no clinician acts differently, the model’s performance is not measurable in outcomes, only in dashboards.
What happens to the patient it gets wrong? Not the error rate — the path. Who notices, how long it takes, and what the recovery costs.
Who receives the output, and what else is competing for their attention? Positive predictive value is a workflow property as much as a statistical one.
Which patients are missing from the training data? And would you be able to tell from the results if the model were failing them specifically?
What would make us stop? Agreed before launch, in writing. A pilot with no stopping rule is not an evaluation.

None of these require a better model. They require deciding what the question is, which is cheaper to do before the pilot than after it.

I work on this with clinical AI teams, and with health systems standing up AI governance.

Book a discovery meeting

Sources

Figures are quoted from the papers linked above. The 2024 Nature Medicine analysis has since carried an author correction; check it before reusing the percentages.