The two questions a board should ask before approving an AI pilot.

Most AI pilots fail before launch because the wrong questions are asked at the wrong altitude. Here are the two that separate signal from theatre.

Boards are now asked to approve clinical AI pilots at nearly every meeting. The questions usually asked are the wrong ones: How accurate is the model? Who else is using it? What does it cost? Those questions feel like diligence. They are not, because none of them predicts whether the pilot will produce anything real. The published record explains why: a scoping review of randomised trials of AI in clinical practice found that 81% reported positive primary endpoints, yet the endpoints were overwhelmingly diagnostic yield or process measures, mostly in single centres, with little demographic reporting and real doubts about generalisability (Han et al., 2024). Impressive pilots are cheap. Consequences are not.

Two questions do the work the usual ones pretend to do.

Question one: whose work changes, and who owns the decision when the system is wrong?

Not "is the model accurate", but: which roles will act differently on Monday morning, what happens to the work the AI creates rather than removes, and where does accountability sit when the output is wrong in either direction?

This question is decisive because the evidence says deployment, not accuracy, is where value is made or destroyed. The most instructive failure is a highly accurate-sounding system deployed at scale: external validation of a widely implemented sepsis prediction model found it missed 67% of sepsis cases while generating alerts on 18% of all hospitalisations, drowning clinicians in noise (Wong et al., 2021). The most instructive success is its mirror image: a sepsis model that worked at Duke, not because the algorithm was remarkable, but because the institution designed a dedicated rapid-response nurse role around it, with calibrated alert volume and clear authority boundaries between nurse and physician (Sandhu et al., 2020). Same problem, opposite outcomes, and the difference was never the model. It was whether anyone had designed the human side of the system.

If the pilot's sponsor cannot answer whose work changes, the honest translation is: no one has designed the workflow, so the pilot will measure the model and prove nothing about care. A board should also ask what the pilot will do to the people in the loop, because the risk is not hypothetical: automation bias, the tendency to over-rely on automated advice, produces errors of commission and omission and is amplified by exactly the workload and time pressure of routine practice (Goddard, Roudsari and Wyatt, 2011).

Question two: what evidence would make us stop, and who is watching for it after go-live?

Not "what does success look like", which invites theatre, but: what are the pre-agreed failure criteria, at which of the four levels of evidence (operational efficiency, clinician experience, diagnostic or process performance, patient outcomes), and what happens after the pilot ends?

Two facts make this question non-negotiable. First, claims migrate up the evidence ladder unless boards hold the line; workflow speed is routinely presented as if it were outcome benefit, and the trial literature shows patient-relevant outcomes remain the exception rather than the rule (Han et al., 2024). Second, models fail silently after deployment. Clinical AI degrades through dataset shift, as populations, practice patterns and upstream systems drift away from the data the model learned from (Finlayson et al., 2021). A pilot without post-deployment monitoring is not a pilot; it is a launch with extra steps. The emerging discipline of algorithmovigilance exists precisely because performance at go-live guarantees nothing a year later (Embi, 2021).

So the second question forces three commitments before approval: stop criteria written down in advance, a named owner for monitoring after the pilot converts to production, and a decision rule for what would trigger recalibration or withdrawal. If those cannot be stated, the organisation is not ready to run the pilot, whatever the vendor's ROC curve says.

The pattern behind both questions

Both questions are the same question at different moments: has the human-AI system, rather than the algorithm, been designed? Approve pilots where the answer is yes. Send the rest back, not as a rejection but as a design brief. A pilot that cannot answer these two questions will succeed as a demonstration and fail as care, and the board will have paid to learn nothing.

References

1. Embi, P.J. (2021) 'Algorithmovigilance: advancing methods to analyze and monitor artificial intelligence-driven health care for effectiveness and equity', JAMA Network Open, 4(4), e214622. https://doi.org/10.1001/jamanetworkopen.2021.4622
2. Finlayson, S.G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I.S. and Saria, S. (2021) 'The clinician and dataset shift in artificial intelligence', New England Journal of Medicine, 385(3), pp. 283-286. https://doi.org/10.1056/NEJMc2104626
3. Goddard, K., Roudsari, A. and Wyatt, J.C. (2011) 'Automation bias: a systematic review of frequency, effect mediators, and mitigators', Journal of the American Medical Informatics Association, 19(1), pp. 121-127. https://doi.org/10.1136/amiajnl-2011-000089
4. Han, R., Acosta, J.N., Shakeri, Z., Ioannidis, J.P.A., Topol, E.J. and Rajpurkar, P. (2024) 'Randomised controlled trials evaluating artificial intelligence in clinical practice: a scoping review', The Lancet Digital Health, 6(5), pp. e367-e373. https://doi.org/10.1016/S2589-7500(24)00047-5
5. Sandhu, S., Lin, A.L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A.D., Balu, S., O'Brien, C. and Sendak, M.P. (2020) 'Integrating a machine learning system into clinical workflows: qualitative study', Journal of Medical Internet Research, 22(11), e22421. https://doi.org/10.2196/22421
6. Wong, A., Otles, E., Donnelly, J.P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M. and Singh, K. (2021) 'External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients', JAMA Internal Medicine, 181(8), pp. 1065-1070. https://doi.org/10.1001/jamainternmed.2021.2626