In Healthcare, AI Governance Is Patient Safety

Healthcare AI governance is patient safety work: deciding which models may touch care, checking them on your own patients, and watching them until they are retired. The hospital that switches a model on owns that safety case. The vendor's figures don't transfer to your patients, and neither does a regulator's clearance.

The clearest evidence is a 2021 study from Michigan Medicine. Researchers ran Epic's sepsis model, used at hundreds of hospitals, across 38,455 hospitalizations and found that it missed two in three patients who developed sepsis. Epic had reported far better numbers. The gap came to light because the hospital's own researchers went looking.

My position will cost money, which is why some committees will resist it. Treat every clinical model you buy as unvalidated until it has been tested on your own records, by people who don't work for the vendor. The evidence for that rule comes first, then the law, then the places patient data leaks.

What went wrong with the Epic sepsis model?

Andrew Wong and colleagues at the University of Michigan studied 27,697 adult patients admitted between December 2018 and October 2019. Sepsis occurred in 2,552 hospitalizations, about 7 percent. At the alert threshold the hospital used, the Epic Sepsis Model flagged a third of those cases. It also alerted on 6,971 hospitalizations in all, so most alarms went to patients who never developed sepsis.

The model's area under the curve was 0.63 at the hospitalization level. Epic's internal documentation, which the authors were allowed to cite, had put it between 0.76 and 0.83, and a public study coauthored with Epic had reported 0.73. Only 12 percent of the alerts went to a patient who had sepsis, which the authors described as "a large burden of alert fatigue."

A second Michigan study in 2024 asked a sharper question. Did the model spot sepsis before clinicians did? Fahad Kamran and colleagues restricted the analysis to predictions made before any sign that a clinician suspected sepsis, such as a blood culture or an antibiotic order. The model's AUROC fell from 0.87 to 0.47, which the authors called "no better than random." Before a blood culture alone, it was 0.53. Their warning applies to every clinical model: inputs such as tests and treatments "could inadvertently encode clinical suspicion," so a model can look strong in a retrospective study while adding little before a clinician recognises the case.

Epic responded. In October 2022 STAT reported that Epic was now recommending that hospitals train the model on their own data before clinical use, had moved to a more widely accepted definition of sepsis onset, and was relying less on clinicians' antibiotic orders to flag the condition. Those are the right changes. A recommendation still leaves the hospital to do the local training and the testing that comes after it.

What would have caught the problem earlier is plain enough: a local check before go-live, an independent look at the vendor's figures, a count of alert burden, and routine monitoring. None of it was required of the hospitals that bought the model. The same organisational pattern shows up well outside medicine, from Watson for Oncology to Air Canada's chatbot. I set out seven versions of it in Most AI Projects Are Failed by the Organisation, Not the Model.

How does bias get into a clinical algorithm?

Usually through a choice nobody thought of as a fairness decision. In 2019 Ziad Obermeyer and colleagues studied a commercial algorithm that health systems used to pick patients for extra care management. It excluded race. It predicted next year's health costs and treated cost as a stand-in for need. Because less money is spent on Black patients with the same illnesses, the algorithm read them as healthier than they were.

At the 97th percentile of risk score, where patients were enrolled automatically, Black patients had 26.3 percent more chronic illnesses than White patients with the same score. In the authors' simulation, fixing the disparity would have raised the share of Black patients selected for extra help from 17.7 percent to 46.5 percent. The paper does not name the vendor. The Washington Post identified it as Optum.

The ending is the useful part. The manufacturer worked with the researchers, and a new label that combined predicted health with predicted cost cut the bias by 84 percent on the company's national dataset of almost 3.7 million patients. The model was fixable once someone outside the vendor looked at which outcome it had been told to predict.

Bias also enters through the instruments. Michael Sjoding and colleagues compared pulse oximeter readings with arterial blood gases at Michigan in 2020. Among readings of 92 to 96 percent, true oxygen saturation was below 88 percent in 11.7 percent of measurements from Black patients and 3.6 percent from White patients, "nearly three times the frequency." Any early warning score that takes oximeter readings as input inherits that error. The FDA notes that oximeter accuracy "can be affected by, among other factors, a person's skin pigmentation", yet its January 2025 guidance on how to test the devices is still a draft.

US health programs that receive federal financial assistance from HHS have had a legal duty here from 1 May 2025. Section 92.210 of the Section 1557 rule requires covered entities to make reasonable efforts to identify patient care decision support tools that "employ input variables or factors that measure race, color, national origin, sex, age, or disability," and to mitigate the risk of discrimination from those tools. Above both duties sits a general ban on discrimination through any such tool, whatever its inputs. My reading is that the identify-and-mitigate duty, keyed to input variables, wouldn't have flagged the Obermeyer algorithm, which used no race variable. The general ban would still apply, which is why you test the outcomes by group as well as the inputs. Whether your data is fit for that work is a separate question, which I take up in Only Seven Percent Say Their Data Is Completely Ready for AI.

Which rules apply to clinical AI in 2026?

The FDA's list of AI-enabled devices, updated in September 2026, holds 1,614 entries by my count (FDA does not print a total). Clearance tells you the FDA accepted the maker's evidence for a stated use. It tells you nothing about your patients. The table sets out what each main rule asks of a hospital or a health-tech company; on phones it scrolls sideways.

Three rows carry more than they show. The January 2026 decision support guidance gives sepsis alerting as its own example of a device function, because "an alarm which supports time-critical decision-making" leaves no time for a clinician to review the basis. The HIPAA Security Rule proposal of January 2025 states that ePHI "in AI training data, prediction models, and algorithm data" is protected, though the rule itself remains a proposal. And under the EU AI Act, the Digital Omnibus moved the high-risk date for AI built into regulated products such as medical devices to 2 August 2028. Dates for other jurisdictions are in How 8 Countries Regulate AI in 2026.

Where can patient data leak from an AI system?

HIPAA programmes were built for databases. An AI system holds patient data in more places than a database does, and some of them are hard to see. I use a five-part map, the PHI Surface Area model, to make those places visible. Each surface rests on published research, shown in the last column (on phones, scroll the table sideways).

Newer, larger models trained on raw records tell a different story from the early research. In 2021 Eric Lehman and colleagues found that simple probes could not get BERT trained on MIMIC-III clinical notes to link inserted patient names to their conditions, while warning that "more sophisticated" attacks might. In June 2026 a study in Nature Communications fine-tuned large language models on more than 13,000 inpatient records from Yale New Haven Health and found memorisation "prevalent and significantly higher than that in the general domain." The fine-tuned models regenerated 3,192 PHI instances from 10,000 training records, each checked by hand against the source notes. Those records had not been de-identified, and this was a researchers' test, not an outside attack. The authors still concluded that de-identification "may still not be sufficient."

The encoding surface is the one I see underrated most. John Morris and colleagues recovered 92 percent of 32-token texts exactly from their embeddings, and 89 percent of full names from clinical notes in which fake names had been inserted. A vector store built from patient notes holds patient data, whatever its schema says. Inference logs are simpler still: if clinicians type patient details into a prompt, the log holds PHI, and it needs the same protection as the record. NIST's generative AI profile names the same risks, from "data memorization" to models that "correctly infer PII or sensitive data" by stitching sources together.

Give each surface an owner, then a control you can actually test. A business associate agreement that covers only ingestion leaves the other four open. The fourth surface is the subject of The HIPAA Review Passed. The Model Remembered Patient Details, and I apply the model to bought software in The AI That Hurts You Is Not the AI You Own.

What does governance look like across a model's life?

The Joint Commission and CHAI guidance of September 2025 lists seven elements: governance structures, privacy and transparency, data security, ongoing quality monitoring, blinded reporting of AI safety events, risk and bias assessment, and education. It is voluntary. Turned into work, the elements fall into four stages.

• Before go-live. Run the model silently on your own recent records and compare it with outcomes, overall and for each patient group. Check what label it predicts and whether its inputs encode clinical suspicion. Classify it against FDA, HIPAA, Section 1557 and state rules. Ask the vendor for its validation data, its change control plan and its figures by subgroup.
• At go-live. Give the patient disclosures that Texas and California now require. Train clinicians on what the tool cannot do. Set the alert threshold with the people who will receive the alerts, and write down how they override the model and where overrides are recorded.
• In use. Monitor accuracy, alert volume, overrides and subgroup gaps on a fixed schedule. Models drift: in one study, 91 percent of 128 model and dataset pairs, including hospital operations data, lost accuracy over time. Name one person who can pause the model and the thresholds that oblige them to.
• At retirement. Decide in advance what ends a model: sustained underperformance, a better validated alternative, a vendor change you did not agree to, or a rule it cannot meet. Keep the records of what it recommended, because the liability trail outlasts the model.

The drift study drew on hospital operations, transport, finance and weather data, so it says nothing directly about clinical outcomes. It does say that a model left alone will rarely stay as good as it was on launch day. For the question of when to stop, see Nobody Tells You When to Stop an AI Deployment. A health system starting from little can begin with minimum viable governance: an inventory, a risk register and a named owner for each model.

None of this is free, and the objection I hear most is fair: a small hospital has no data science team and little leverage over a large vendor. Three things lower the cost. Start with the models that drive time-critical alerts, where a silent error does the most harm. Share the testing through a network, an academic partner or a group of peer hospitals, since a check on similar patients beats none. And write the duties into the contract: the right to test and publish results, performance figures by patient group, notice before the model changes, and audit rights. The hospitals least able to pay for validation often serve the patients most exposed to the biases above, which is a reason to share the work, not to skip it.

Questions for the AI committee before go-live

A committee can run these in one meeting for any clinical model. If an answer is missing, the model waits, or runs in silent mode until the answer is in.

1. Has this model been tested on our own patients, by people who do not work for the vendor, and how did it do for each patient group?
2. What outcome does it predict, and could that outcome stand in for something unequal, such as cost or access?
3. Do its inputs include tests or treatments that only appear once a clinician already suspects the condition?
4. How many alerts will it send a clinician per shift, and how many of them will be right?
5. Is it a device under FDA rules, a tool under Section 1557, or high-risk under the EU AI Act?
6. Which of the five PHI surfaces does it touch, and who owns each one?
7. Who can pause it today, what numbers oblige them to, and when did we last check those numbers?

Michigan's researchers found the sepsis model's problems because they went looking. By then the model was already in use at hundreds of hospitals. The point of governance in a hospital is to make that search routine, and to do it before the patient is in the bed.

The financial services version of this argument, where the cost of failure is measured in money, is in Nine Regulators, One Control Set for Bank AI.

Sources

• Andrew Wong, Erkin Otles, John P. Donnelly and others, "External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients", JAMA Internal Medicine 181(8), published online 21 June 2021.
• Fahad Kamran, Donna Tjandra, Andrew Heiler and others, "Evaluation of Sepsis Prediction Models before Onset of Treatment", NEJM AI 1(3), 7 February 2024.
• Casey Ross, "Epic overhauls popular sepsis algorithm criticized for faulty alarms", STAT, 3 October 2022.
• Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, "Dissecting racial bias in an algorithm used to manage the health of populations", Science 366, 25 October 2019.
• Carolyn Y. Johnson, "Racial bias in a medical algorithm favors white patients over sicker black patients", Washington Post, 24 October 2019.
• Michael W. Sjoding, Robert P. Dickson, Theodore J. Iwashyna and others, "Racial Bias in Pulse Oximetry Measurement", New England Journal of Medicine 383, 17 December 2020.
• US Food and Drug Administration, "Pulse Oximeters for Medical Purposes", draft guidance, 7 January 2025.
• US Department of Health and Human Services, 45 CFR 92.210, "Nondiscrimination in the use of patient care decision support tools", final rule 89 FR 37522, 6 May 2024.
• US Food and Drug Administration, Artificial Intelligence-Enabled Medical Devices, list updated 4 September 2026.
• US Food and Drug Administration, "Clinical Decision Support Software", guidance, January 2026.
• US Food and Drug Administration, "Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions", final guidance, 4 December 2024, reissued 18 August 2025.
• 45 CFR 160.103 (business associate) and 45 CFR 164.514 (de-identification).
• US Department of Health and Human Services, "HIPAA Security Rule To Strengthen the Cybersecurity of Electronic Protected Health Information", proposed rule, 90 FR 898, 6 January 2025.
• Regulation (EU) 2024/1689 (the AI Act), Article 6(1) and Annex I, as amended by Regulation (EU) 2026/1744, Official Journal, 24 July 2026.
• Texas HB 149 (Business and Commerce Code section 552.051) and SB 1188 (Health and Safety Code section 183.005), 2025.
• California Health and Safety Code section 1339.75 (AB 3030, 2024) and AB 489 (2025).
• The Joint Commission and Coalition for Health AI, "The Responsible Use of AI in Healthcare", September 2025.
• Anran Li and others, "Memorization in large language models in medicine: prevalence, characteristics, and implications", Nature Communications 17, June 2026.
• Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg and Byron C. Wallace, "Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?", NAACL 2021.
• John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov and Alexander M. Rush, "Text Embeddings Reveal (Almost) As Much As Text", EMNLP 2023.
• Nicholas Carlini, Florian Tramèr, Eric Wallace and others, "Extracting Training Data from Large Language Models", USENIX Security 2021.
• Milad Nasr, Nicholas Carlini and others, "Scalable Extraction of Training Data from (Production) Language Models", arXiv, 28 November 2023.
• Reza Shokri, Marco Stronati, Congzheng Song and Vitaly Shmatikov, "Membership Inference Attacks Against Machine Learning Models", IEEE Symposium on Security and Privacy, 2017.
• Luc Rocher, Julien M. Hendrickx and Yves-Alexandre de Montjoye, "Estimating the success of re-identifications in incomplete datasets using generative models", Nature Communications 10, 23 July 2019.
• National Institute of Standards and Technology, AI 600-1, Generative Artificial Intelligence Profile, July 2024.
• Daniel Vela, Andrew Sharp, Richard Zhang and others, "Temporal quality degradation in AI models", Scientific Reports 12, 8 July 2022.

All sources checked on 6 October 2026.

Changes on 6 October 2026. Rewritten from about 5,000 words as a Reference piece with a new headline. The Epic figures now give their setting (one external validation at Michigan Medicine) and add the 2024 before-treatment analysis. The "200 million people" figure for the Optum algorithm is gone: the study applies it, by industry estimate, to the whole class of such tools. The PHI Surface Area model moved here from the HIPAA piece, with a published source for each surface. Added: Section 1557, the January 2026 FDA decision support guidance, and Texas and California rules. Removed: market-size and adoption figures, the six use-case profiles, the framework mapping, and a download box for a checklist that did not exist.