Human-in-the-loop is not a safety strategy

Human-in-the-loop has become the comfort blanket of healthcare AI governance. Almost every framework, every regulator and every vendor invokes it. The difficulty is that it is doing far less work than people assume. Positioning a clinician as the checkpoint at the end of an algorithm feels like safety. Often it is only the appearance of safety.

Our argument is a reframing. The governance object is not the model, and it is not the reviewer. It is the combined human-AI work system. The test is not whether a human checks the output. It is whether the joint team is safer, fairer, faster and more accountable than the workflow that came before it.

Why the checkpoint fails

Start with the clearest evidence that a human in the loop does not, by itself, make a system safe. When the widely deployed Epic Sepsis Model was externally validated across nearly 40,000 hospitalisations, it missed 67% of sepsis cases while generating alerts on 18% of all hospitalisations, a large burden of alert fatigue with poor discrimination (Wong et al., 2021). The human-in-the-loop design did not rescue that system. It operationalised noise, and asked clinicians to sift it.

The reason checkpoints degrade is well documented. Automation bias, the tendency to over-rely on automated advice, is not a corner case; a systematic review found it produces both errors of commission and of omission, amplified precisely by workload, time pressure and miscalibrated trust, the conditions under which clinical AI is most often deployed (Goddard, Roudsari and Wyatt, 2011). A second review found that the harder an output is to verify, the more the reviewer simply defers, and that this holds even in single-task settings with high cognitive load (Lyell and Coiera, 2017). Agentic systems sharpen the problem, because they do not only suggest, they act, reducing the moments at which scrutiny is natural.

There is a deeper irony. The more reliable the system becomes, the worse the human gets at catching its rare failures, because genuine error detection becomes a low-base-rate signal task and clinical skill atrophies through disuse. The safety mechanism weakens exactly as the technology it guards improves. No governance framework built on the checkpoint alone has solved this.

What actually worked

Contrast the failures with a deployment that held. Duke’s sepsis early-warning programme succeeded not because a human sat in the loop, but because the institution designed a specific role around the model: a rapid-response nurse with defined competence, an alert volume calibrated to that role’s capacity, and clear communication protocols between nurse and physician (Sandhu et al., 2020). Role design did the work, not oversight in the abstract.

That is the pattern. The strongest deployments treat the human-AI combination as the unit of performance and build role design, alert budgets, learning loops and continuous monitoring around it. The weakest rely on the existence of a checkpoint. And the monitoring is not optional: models fail silently after deployment through dataset shift, as clinical populations and inputs drift away from the data a model was trained on (Finlayson et al., 2021). Detecting that requires active post-deployment surveillance, an emerging discipline some now call algorithmovigilance (Embi, 2021), not a reviewer glancing at one case at a time.

From checkpoint to teaming

If the object of governance is the team, six things have to be designed, not assumed. Task allocation: which parts of the cognitive work the AI does, at what level of automation, by risk tier. Human role design: the accountable role’s credentials, authority, training, time budget, override rights and deactivation triggers, treated as a clinical operating role rather than a disclaimer. A collaboration protocol: when the human sees the output, what uncertainty is surfaced, and what happens on disagreement. A safety envelope: thresholds and alert budgets calibrated to the human role’s real capacity, not only to the model’s ROC curve. A learning loop: overrides, disagreements, misses and downstream workload logged and fed back, because disagreement between clinician and model is data, not deviance. And governance: a living clinical safety case, change control, incident response and an explicit authority to switch the system off.

An organisation that deploys agentic AI without redesigning those six layers is not implementing AI. It is bolting AI onto a workflow that was never built for it, and that is where most failure modes come from.

Why we still keep humans

None of this is an argument for removing the clinician. It is an argument for being honest about why the clinician is there. The human is not in the loop to rubber-stamp the machine. The human is in the loop because clinical care requires judgement in cases that have no textbook answer, advocacy for the patient with the family, the institution and the payer, and the willingness to accept professional responsibility for the outcome. The wider scientific community has declined to grant AI authorship for exactly that last reason: a model can generate text, but it cannot accept accountability.

Regulation is beginning to catch up. Good Machine Learning Practice, co-published by the US FDA, Health Canada and the UK MHRA, already directs attention towards human-AI team performance rather than model performance alone (US FDA, Health Canada and MHRA, 2021), and Article 14 of the EU AI Act requires human oversight that is genuinely effective, including the ability to interpret outputs and override the system (European Union, 2024). Both tell you to do oversight. Neither tells you how to do it well. That is the work.

Capability without governance is not resilience. It is exposure (del Río, 2026). The same is true of oversight. A checkpoint without role design, competence standards, alert budgets, learning loops and accountability is not oversight. It is theatre with a clinician standing in it.

References

1. del Río, A. (2026) ‘Crossing the Rubicon in the age of agentic AI: clinical risk, exposure and cost in digital health’, HealthManagement, 26(3).
2. Embi, P.J. (2021) ‘Algorithmovigilance: advancing methods to analyze and monitor artificial intelligence-driven health care for effectiveness and equity’, JAMA Network Open, 4(4), e214622. https://doi.org/10.1001/jamanetworkopen.2021.4622
3. European Union (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Article 14. Official Journal of the European Union.
4. Finlayson, S.G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I.S. and Saria, S. (2021) ‘The clinician and dataset shift in artificial intelligence’, New England Journal of Medicine, 385(3), pp. 283-286. https://doi.org/10.1056/NEJMc2104626
5. Goddard, K., Roudsari, A. and Wyatt, J.C. (2011) ‘Automation bias: a systematic review of frequency, effect mediators, and mitigators’, Journal of the American Medical Informatics Association, 19(1), pp. 121-127. https://doi.org/10.1136/amiajnl-2011-000089
6. Lyell, D. and Coiera, E. (2017) ‘Automation bias and verification complexity: a systematic review’, Journal of the American Medical Informatics Association, 24(2), pp. 423-431. https://doi.org/10.1093/jamia/ocw105
7. Sandhu, S., Lin, A.L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A.D., Balu, S., O’Brien, C. and Sendak, M.P. (2020) ‘Integrating a machine learning system into clinical workflows: qualitative study’, Journal of Medical Internet Research, 22(11), e22421. https://doi.org/10.2196/22421
8. US Food and Drug Administration, Health Canada and Medicines and Healthcare products Regulatory Agency (2021) Good Machine Learning Practice for Medical Device Development: Guiding Principles.
9. Wong, A., Otles, E., Donnelly, J.P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M. and Singh, K. (2021) ‘External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients’, JAMA Internal Medicine, 181(8), pp. 1065-1070. https://doi.org/10.1001/jamainternmed.2021.2626