Key Takeaways
- →Models fine-tuned on patient notes can reproduce them. In a June 2026 Yale study, one regenerated 3,192 PHI items, as the study defined PHI, from 10,000 training records that hadn't been de-identified.
- →HIPAA defines PHI by what it identifies, in any form or medium. Patient details in a chatbot's reply count.
- →HHS says ePHI in AI training data and prediction models is protected, and expects AI in the risk analysis. The Security Rule update that says so is still a proposal, with a final rule targeted for July 2027.
- →The rules in force already reach a trained model: risk analysis, minimum necessary, de-identification, the business associate contract and the breach presumption.
- →Encryption and a signed business associate agreement don't test the model's answers. A memorisation probe does, and its result belongs in the risk analysis.
Memorisation is what happens when a model trained on records reproduces parts of them in its answers. Train one on patient notes and it can hand parts of them back when prompted with the opening of a record it saw. Under HIPAA, what comes back is still protected health information.
My position: treat any model fine-tuned on patient records as a store of those records until a memorisation test finds a rate your risk owner has signed off, and name that test in the HIPAA risk analysis. That is a handling rule. It makes no claim about how a regulator or a court would classify the weights. Some counsel will say weights are numbers, not records; I wouldn't bet a patient's privacy on it. This piece is for the HIPAA privacy officer who signs off the model before it goes live.
The scene below is a composite of a common pattern, not a hospital I can name. A health system fine-tunes a model on its own clinical notes. The vendor has signed a business associate agreement, the training data is encrypted, access is logged, and the compliance review comes back clean. Nobody has tested what the model itself now holds. A study published in June 2026 shows what such a test can find.
Can an AI model leak the patient records it was trained on?
Anran Li, Qingyu Chen and colleagues at Yale reported in Nature Communications on how often medical language models memorise their training data. In one test they fine-tuned Llama-3-Instruct-8B, an open general-purpose model, to suggest diagnoses from 10,000 inpatient records of Yale New Haven Health, holding about 3,000 more back for validation and testing. Fine-tuning made the model better at diagnosis: top-1 accuracy rose from 48.6 to 54.8 percent. It also made the model remember.
The team gave the model part of each training record and compared what it wrote next with the original. Depending on the model and the test, 2 to 30 percent of outputs repeated at least 30 tokens of the original word for word. Across the 10,000 records, the fine-tuned Llama model regenerated 3,192 instances of PHI, each checked by hand against the patient's notes. A manual review of 200 outputs found 30 more that the automated detector had missed, and 98 sensitive details outside the PHI definitions, such as diagnoses, family relationships and treatment details.
Read the limits before you quote the number. The records were not de-identified, by design, so the team could see what the model kept. Their PHI definition starts from HIPAA's 18 identifiers and adds indirect ones, such as hospital and clinician names. This was a researchers' test with the training records in hand, and the authors say they did no targeted testing of how much an attacker could extract. The paper gives no count of the patients behind the 3,192 items. The authors still concluded that de-identification "may still not be sufficient", because automated tools miss identifiers and sensitive details fall outside the definitions. Practitioners, they wrote, "could consider evaluating memorization in addition to accuracy-based metrics."
Does HIPAA cover what a model memorises?
The rule text never mentions models, but its definitions reach far. Protected health information is individually identifiable health information, whether held in electronic media "or in any other form or medium" (45 CFR 160.103). A model output that names a patient and a diagnosis meets that test. Whether the weights themselves maintain PHI is a question the regulation does not answer.
HHS has said what it thinks. In the preamble to its January 2025 proposal to update the Security Rule, it wrote that ePHI "in AI training data, prediction models, and algorithm data" maintained for covered functions "is protected by the HIPAA Rules". It also expects an entity using AI to "include the use of such tools in its risk analyses." That is the department's stated view, not rule text, and the proposal is not final. The federal regulatory agenda now lists it as a long-term action, with a final rule targeted for July 2027.
What does HIPAA already require of a model trained on patient data?
Plenty, without the update. Five provisions bear on such a model. The table gives my reading of what each means for the model, then the text itself; on phones it scrolls sideways.
| Provision (45 CFR) | What it means for a trained model | What the text requires |
|---|---|---|
Risk analysis, 164.308(a)(1)(ii)(A) | Memorisation is a confidentiality risk. Record how it was tested | An "accurate and thorough assessment" of risks to the ePHI the entity holds |
Minimum necessary, 164.502(b) | Train on the fields the task needs, not the whole chart | Reasonable efforts to limit PHI to the minimum necessary for the purpose |
De-identification, 164.514(b) | Automated scrubbing of notes misses identifiers. Check what is left | An expert finds the risk "very small", or 18 identifiers are removed and the entity has no actual knowledge the rest could identify anyone (Safe Harbor) |
Business associate contract, 164.504(e)(2)(ii)(J) | Settle what happens to weights trained on your records | At termination, return or destroy PHI the vendor "still maintains in any form" if feasible; if not, extend the protections to it |
Breach, 164.402, 164.404 | An output shown to the wrong person may qualify. Log outputs so you can tell | An impermissible disclosure is presumed a breach. Notify without unreasonable delay, within 60 days of discovery |
Three rows need a closer look. First, the risk analysis is mandatory, and it covers risks to the ePHI you hold. Your training set is that ePHI, and a model that can regenerate it is one way for it to leave. That puts the probe in scope, whatever the weights are.
Encryption is an "addressable" safeguard under 164.312, which means implement it or document why an equivalent measure is reasonable. It protects stored and transmitted data, and it does nothing about what the model says when asked. Its main payoff in the breach rule is that notification duties attach to "unsecured" PHI, meaning PHI not rendered unreadable by a method HHS specifies. A model's plain-text answer is readable by design.
The termination clause was written for databases. If your vendor fine-tuned the model on your records and it can regenerate them, my reading, which some counsel will contest, is that the vendor still maintains your PHI in some form when the contract ends. Write into the contract whether those weights are destroyed or stay under its protections.
The breach row has a catch. Under 164.402, an impermissible use or disclosure is presumed a breach unless a four-factor risk assessment shows a low probability of compromise, and some good-faith disclosures inside the organisation are excluded. Whether an answer is reportable depends on who saw it. The 60-day clock runs from discovery, and a breach counts as discovered when reasonable diligence would have found it, so not logging outputs does not stop the clock. A regenerated diagnosis with no identifier attached may never count as a breach. Put it in the risk analysis anyway, since the patient's exposure does not depend on the definition.
What should a health system test before go-live?
Five checks, and none of them waits for the new rule.
- Probe for memorisation. Before release and after every retraining, prompt the model with the opening of training records and compare what it writes with the rest, as the Yale team did. Run the same probe on the base model so you know what fine-tuning added, and set the acceptable rate before you start. Count identifiers and sensitive details as well as exact matches.
- Write the result into the risk analysis. Name the test, the date, the rate found and who accepted it. That turns the HHS expectation into a record an auditor can read.
- Cut the training data to the task. Apply minimum necessary field by field, de-identify where the task allows, and check a sample of scrubbed notes by hand. Stop training when validation loss stops improving: in the Yale study, the memorisation rate rose from 7.53 to 11.61 percent when training ran three epochs past that point.
- Fix the contract. Put memorisation testing, output logging and the fate of the weights at termination into the business associate agreement. The wider vendor questions are in The AI That Hurts You Is Not the AI You Own.
- Log and scan outputs. Treat prompt and response logs as PHI, and scan outputs for identifiers, so a control finds a disclosure before a patient does.
Memorisation is one of five places patient data can leak from an AI system. The others are ingestion, encoding, inference and reconstruction. The full map, with the research behind each surface, is in In Healthcare, AI Governance Is Patient Safety. If you offer care to people in the EU, the GDPR applies as well; see The GDPR Was Not Written for AI. It Governs It Anyway.
Go back to the composite health system. Its review was accurate about everything it checked. Nobody put the question to the model. There's now a published method for asking it.
This is a practitioner's reading of the regulation, not legal advice.
Questions readers ask
Is a model trained on patient data PHI under HIPAA?
An output that identifies a patient and their health is PHI, because 45 CFR 160.103 covers identifiable health information held in electronic media "or in any other form or medium". Whether the weights themselves maintain PHI is a question the regulation does not answer. HHS's view, in the preamble to its January 2025 Security Rule proposal, is that ePHI in training data and prediction models is protected. My advice: if PHI went into training, handle the model as holding PHI until a memorisation test finds a rate your risk owner has signed off.
Can a hospital train an AI model on patient records under HIPAA?
Yes, with a permitted purpose. Quality improvement for the hospital's own care can be "health care operations" under 45 CFR 164.501, provided that "the obtaining of generalizable knowledge is not the primary purpose". Research, which aims at generalisable knowledge, needs patient authorisation or a waiver from an IRB or privacy board (164.512(i)). Minimum necessary applies either way, and a vendor that trains the model on your behalf is a business associate.
Is the HIPAA Security Rule update final?
No. HHS proposed it on 6 January 2025 (90 FR 898). The federal regulatory agenda now lists it as a long-term action and gives July 2027 as the target for a final rule. Until then the current Security Rule applies, including its risk analysis requirement.
Get The Brief
Original thinking on AI strategy and governance. Every other Tuesday.

Senior AI strategist helping leaders make AI real across four continents. Forbes Technology Council member, IEEE Senior Member.
Ajay's views, from fifteen-plus years in the field. Not legal or compliance advice. See full disclaimers →
Published by AI Exponent LLC



