AI governance does not live in a policy document.

Governance is what happens at the point of use, not the point of approval. A short note from the floor.

Ask a health system to show you its AI governance and you will usually be shown a document: a policy, a committee terms of reference, an approval workflow with a risk matrix. All necessary. None of it is governance. A policy describes how the organisation intends to behave. Governance is how the system actually behaves at 11:40 on a Tuesday, when the alert fires for the fourth time in an hour, the ward is short, and a tired clinician has eight seconds to decide whether the machine is right.

The distinction is not rhetorical. It is where the failures live.

Approved, overseen, and still wrong

The clearest demonstration is a system that had passed every document-level test: procured, approved, deployed at hundreds of hospitals, with clinicians in the loop reviewing every alert. External validation of that widely implemented sepsis model found it missed 67% of sepsis cases while alerting on 18% of all hospitalisations (Wong et al., 2021). Every governance artefact existed. What did not exist was governance where it mattered: nobody owned the question of whether the alerts were doing anything but generating noise at the bedside.

The reviewer-as-safeguard assumption fails for well-documented reasons. Automation bias, the human tendency to over-rely on automated advice, produces errors of commission and omission and is amplified by workload, time pressure and low confidence, exactly the conditions of routine care (Goddard, Roudsari and Wyatt, 2011). A clinician positioned as a checkpoint without training, time and authority is not a control. He is a signature.

And even a system that behaves at go-live does not stay governed by the original approval, because the world moves. Clinical AI degrades silently through dataset shift as populations, practice patterns and upstream data change beneath it (Finlayson et al., 2021). The governance decision made at approval describes a system that, months later, no longer exists.

Where governance actually lives

If not in the document, where? In five places, all operational.

In the workflow: who sees the output, when, with what uncertainty attached, and what happens on disagreement. In the role: a named, trained, resourced human function with the authority to act and the time to think, not a disclaimer with a licence number. In the thresholds: alert budgets and escalation triggers calibrated to the receiving team's real capacity rather than to the model's ROC curve. In the monitoring: continuous post-deployment surveillance for drift, silent failure and inequitable performance, the discipline now called algorithmovigilance (Embi, 2021). And in the learning loop: overrides and disagreements logged and reviewed as data, feeding retraining, redesign or withdrawal.

None of these live in a policy document. All of them live in the operating model, which is why governance is an organisational design problem before it is a compliance problem. The regulators, to their credit, are pointing the same way: the Good Machine Learning Practice principles co-published by the US FDA, Health Canada and the UK MHRA direct attention to the performance of the human-AI team, not the model in isolation (US FDA, Health Canada and MHRA, 2021), and Article 14 of the EU AI Act requires human oversight that is effective in use, including the practical ability to interpret and override (European Union, 2024). The regulation tells you that oversight must work. It does not build the workflow, the role, the thresholds, the monitoring or the loop. That is the institution's work, and it is precisely the work the policy document tends to stand in for.

The test

Our colleague put the stake in the ground in her cover story on agentic AI: "Capability without governance is not resilience. It is exposure." (del Río, 2026). The corollary belongs on the wall of every AI committee: governance without operations is not governance. It is paperwork.

The test we apply is simple. Choose any AI system the organisation runs. Ask who owns it this month, what its alert burden was last week, what its override rate is, when its performance was last checked against the population it now serves, and what would trigger switching it off. If those answers exist, governance lives there. If the answer is a document, it does not, whatever the document says. Oversight by design means designed into the work, and nowhere else.

References

1. del Río, A. (2026) 'Crossing the Rubicon in the age of agentic AI: clinical risk, exposure and cost in digital health', HealthManagement, 26(3). healthmanagement.org
2. Embi, P.J. (2021) 'Algorithmovigilance: advancing methods to analyze and monitor artificial intelligence-driven health care for effectiveness and equity', JAMA Network Open, 4(4), e214622. https://doi.org/10.1001/jamanetworkopen.2021.4622
3. European Union (2024) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), Article 14. Official Journal of the European Union.
4. Finlayson, S.G., Subbaswamy, A., Singh, K., Bowers, J., Kupke, A., Zittrain, J., Kohane, I.S. and Saria, S. (2021) 'The clinician and dataset shift in artificial intelligence', New England Journal of Medicine, 385(3), pp. 283-286. https://doi.org/10.1056/NEJMc2104626
5. Goddard, K., Roudsari, A. and Wyatt, J.C. (2011) 'Automation bias: a systematic review of frequency, effect mediators, and mitigators', Journal of the American Medical Informatics Association, 19(1), pp. 121-127. https://doi.org/10.1136/amiajnl-2011-000089
6. US Food and Drug Administration, Health Canada and Medicines and Healthcare products Regulatory Agency (2021) Good Machine Learning Practice for Medical Device Development: Guiding Principles.
7. Wong, A., Otles, E., Donnelly, J.P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M. and Singh, K. (2021) 'External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients', JAMA Internal Medicine, 181(8), pp. 1065-1070. https://doi.org/10.1001/jamainternmed.2021.2626