Key Takeaways
- →LLMs trained on clinical notes can regurgitate patient records: a breach, not a theory
- →Five PHI surfaces exist in AI systems; traditional compliance covers only one
- →Inference logs almost always contain PHI and are almost never secured to HIPAA standards
- →Most AI vendor BAAs do not address model memorization, embedding storage, or adversarial testing
Last September I walked into a compliance review at a mid-size health system in the Midwest. They'd deployed a clinical decision support tool: RAG-based, pulling from internal clinical guidelines and drug interaction databases. Smart architecture. Clean interface. The compliance team had checked every traditional HIPAA box: BAA signed, data encrypted, access controls in place. They were proud of it.
I asked one question: 'Have you tested whether the model memorises patient data from its training set?' Silence. Nobody in the room had considered that an LLM trained on clinical notes might learn to reproduce fragments of those notes verbatim, including patient identifiers. We ran a memorisation probe that afternoon. The model could regurgitate partial patient records when prompted with specific clinical scenarios. That's not a theoretical vulnerability. That's a breach.
The Stakes Are Not Abstract
Healthcare has led all industries in breach costs for fourteen consecutive years, averaging $7.42 million per breach in IBM's most recent Cost of a Data Breach Report. That's not a coincidence. It's a reflection of the sensitivity of the data and the severity of the regulatory regime. But here's what that number obscures: AI systems create breach vectors that traditional HIPAA compliance was never designed to detect. The HHS breach portal logged over 725 large breaches (500 or more individuals affected) in 2023. Not one was classified as an AI-related breach. That doesn't mean AI isn't involved. It means we're not looking.
I need to say this directly: most organisations treating HIPAA AI compliance as a checkbox exercise are building on sand. Signing a BAA and encrypting your database doesn't cover model memorisation, embedding leakage, inference log exposure, or adversarial extraction attacks. These aren't edge cases. They're the default attack surface of any AI system that touches PHI. NIST's Generative AI Profile, a companion to the AI Risk Management Framework, names data memorisation and training-data leakage as a specific risk category. Your compliance programme should too.
The Compliance Iceberg: What Your Audit Misses
Every HIPAA compliance audit I've reviewed for AI systems has the same blind spot. They assess what's visible (BAAs, encryption, access controls) and miss the AI-specific risks hiding below the waterline. This isn't negligence. It's structural. Traditional compliance frameworks weren't built for systems that learn, memorise, and generalise from data.
The iceberg metaphor isn't dramatic licence. In my experience advising healthcare AI deployments, the risks below the waterline are where enforcement actions will come from over the next three years. OCR's proposed update to the HIPAA Security Rule already treats ePHI used in AI training data and algorithms as squarely in scope, and the organisations that haven't built their compliance architecture for these risks will be caught flat-footed. If you haven't assessed your broader AI governance readiness, the 5-Pillar AI Readiness Assessment is where I'd start. The governance pillar maps directly to HIPAA AI compliance requirements.
HIPAA's Three Rules Through an AI Lens
HIPAA was enacted in 1996. Before machine learning. Before cloud computing. Before large language models. The regulation's core concepts (protected health information, covered entities, business associates) are sound. But their application to AI demands careful reinterpretation, not rote compliance.
HIPAA's Three Rules: AI-Specific Interpretation
Each rule creates unique challenges for AI systems
The Privacy Rule's Minimum Necessary standard is fundamentally at odds with how machine learning works. ML models perform better with more data. HIPAA says use less. Resolving this tension requires purpose-limitation architecture: AI systems designed to request, access, and process only the specific data elements needed for each function, at the feature level, not the system level. Every model training run must document which PHI fields were accessed and justify each against the minimum necessary standard. This isn't optional. It's the rule. The practical implication: you need data access controls that are granular enough to distinguish between 'this model needs diagnosis codes' and 'this model needs the full clinical note.' Most healthcare organisations don't have that granularity. Building it is step one.
Mapping PHI Flow Through Your AI Pipeline
Before you can secure PHI in an AI system, you need to see where it flows. Not in the abstract. In your specific pipeline. Every point where PHI enters, is processed, is stored, or is transmitted is a compliance obligation under the Security Rule. Most organisations can map their traditional data flows. Almost none can map their AI data flows.
The highlighted flows in this diagram are where I find the most compliance gaps. The ingestion-to-training path is obvious: most teams encrypt training data. But the ingestion-to-vector-store path? Embeddings encode semantic meaning from PHI. Whether embeddings are PHI under HIPAA is an open legal question, but recent research demonstrates that text can be partially reconstructed from embeddings, which makes the conservative interpretation clear: treat them as PHI. And inference logs, the detailed record of every query and response, almost always contain PHI and are almost never secured to Security Rule standards.
If you can't draw a complete diagram of every point where PHI enters, is processed, is stored, or leaves your AI system, you can't demonstrate Security Rule compliance. The diagram IS the compliance artefact.
The PHI Surface Area Model
Here's the framework I use with healthcare clients to think about AI-specific HIPAA compliance. I call it the PHI Surface Area Model. Traditional compliance focuses on the 'front door,' where PHI enters the system. AI compliance requires covering the entire surface area: every point where PHI could leak, be memorised, be inferred, or be reconstructed.
- Ingestion Surface: Where PHI enters the AI system. Training data, fine-tuning data, RAG corpus, user queries. Covered by traditional HIPAA compliance.
- Encoding Surface: Where PHI is transformed into model representations. Embeddings, model weights, feature vectors. Not covered by traditional compliance. Requires AI-specific controls.
- Inference Surface: Where PHI appears in model outputs. Generated responses, retrieved documents, inference logs. Partially covered. Inference logs are the biggest gap.
- Memorisation Surface: Where the model has learned to reproduce PHI from training data. Only detectable through active probing. No passive monitoring catches this.
- Reconstruction Surface: Where PHI can be reverse-engineered from model artefacts. Embedding inversion, gradient attacks on federated models, adversarial prompting. Emerging threat. Most organisations are completely unprotected.
The model is simple. Cover all five surfaces, and you've covered the AI-specific HIPAA compliance landscape. Miss any one, and you have an unmonitored breach vector. For the broader governance framework that wraps around this model, see Minimum Viable AI Governance.
The Compliance Gap: Typical vs. AI-Ready
This radar shows the gap between where most healthcare organisations are and where they need to be. The outer polygon represents an AI-ready organisation that has implemented controls across all five PHI surfaces. The inner polygon is a typical healthcare organisation: strong on traditional controls, dangerously weak on AI-specific ones.
The shape of the gap tells the story. Traditional PHI access controls: most organisations score reasonably well. Model memorisation testing, embedding security, and AI-specific BAA clauses are consistently the weakest scores I see across engagements, far behind the traditional controls. These are illustrative estimates drawn from my advisory work, not a published industry survey, but the gap they describe shows up in nearly every organisation I've assessed. These aren't small gaps. They're chasms. And they represent the exact vectors that OCR enforcement actions will target as the regulatory framework catches up to the technology.
Privacy-Enhancing Technologies: The Strategic Landscape
The good news, and there is good news, is that the technology landscape for privacy-preserving AI has matured faster than most compliance teams realise. Four approaches are particularly relevant for reducing the PHI surface area in healthcare AI.
The Maturity-Protection Tradeoff
The matrix reveals a frustrating reality: the technologies with the strongest privacy guarantees are the least mature for enterprise deployment. De-identification is widely adopted but provides limited protection against re-identification attacks. Research published in Nature Communications demonstrated that 99.98% of Americans can be re-identified from 15 demographic attributes. Homomorphic encryption provides mathematical guarantees but remains too computationally expensive for most real-time AI workloads.
My recommendation for most healthcare AI deployments: start with federated learning for model training (keeps PHI local) and synthetic data for development and testing environments. Layer differential privacy on top for production inference. This gives you meaningful protection today without waiting for homomorphic encryption to become practical.
Privacy-Enhancing Technologies: Deep Dive
Enables model training across multiple healthcare organisations without centralising PHI. Each institution trains on local data and shares only model updates: gradients, not records. Particularly powerful for rare disease research where no single institution has sufficient data. Critical HIPAA consideration: model gradients can sometimes be reverse-engineered to extract training data. Apply differential privacy or gradient encryption to the update mechanism. This is non-optional for any federated deployment handling PHI.
Generative AI creates realistic but non-identifiable datasets for training and testing. Reduces reliance on actual PHI while preserving statistical properties needed for model development. Two HIPAA cautions: (1) the generator itself is trained on real PHI, making it a covered system, and (2) generated data must be validated to ensure it doesn't contain identifiable patterns. I've seen synthetic datasets that passed basic de-identification checks but contained enough correlated attributes to enable re-identification. Validate aggressively.
Mathematical guarantees of individual privacy through calibrated noise injection. Prevents any single individual's data from significantly influencing model behaviour. The tradeoff is real: stronger privacy guarantees reduce model accuracy. In healthcare, where clinical accuracy is non-negotiable, calibration requires domain expertise. A 2% accuracy drop might be acceptable for population health analytics. It's not acceptable for diagnostic imaging. Calibrate per use case.
Computation on encrypted data without decryption: PHI stays protected even during processing. Theoretically the gold standard. Practically, current performance limitations restrict applicability to specific, latency-tolerant use cases: secure aggregation, batch analytics, simple classification. Complex AI inference is still 100-1000x slower under homomorphic encryption. Watch this space. Performance is improving by roughly 10x per year.
BAAs Are Not Ready for AI
I'll be direct. Almost every Business Associate Agreement I've reviewed for AI vendors is inadequate. They're written for traditional data processing: database hosting, claims processing, electronic health records. They don't cover the AI-specific obligations that HIPAA compliance demands.
“A BAA that doesn't address model memorisation, embedding storage, inference logging, and adversarial testing is a BAA that doesn't cover your AI system. And a BAA that doesn't cover your system is a compliance gap with your name on it.”
Here's what your AI vendor BAA must include, and what most are missing:
- Model training data handling: What happens to PHI used for training? Is it retained in the model weights? Can the vendor demonstrate that PHI is not memorised? What testing protocols are in place?
- Embedding and vector storage: How are PHI-derived embeddings stored, secured, and eventually deleted? Are embeddings treated as PHI under the agreement?
- Inference log management: How long are inference logs retained? Who has access? Are they encrypted to Security Rule standards? Can individual patient records be purged from logs?
- Memorisation and adversarial testing: Does the vendor commit to periodic memorisation probes? What's the remediation protocol if memorisation is detected?
- Model deletion: When the BAA terminates, what happens to the model? If it was trained on your PHI, simply deleting the training data isn't sufficient. The model itself may encode PHI patterns.
The 90-Day Compliance Roadmap
Theory doesn't protect patients. Here's the phased approach I use with healthcare clients to close AI-specific HIPAA compliance gaps in 90 days.
Detailed Implementation Plan
Each phase builds on the previous
Days 1–14: AI System Audit
Inventory every AI application that touches PHI, including tools your clinical staff adopted without IT approval. For each system, map the PHI flow: where does PHI enter, how is it processed, where is it stored, how does it appear in outputs? Document which systems have BAAs and which are operating in regulatory grey areas. The output is a PHI Surface Area Map for each AI system.
Days 15–35: BAA Modernisation
Review every AI vendor BAA against the five-point checklist above. Prioritise vendors whose systems have the largest PHI surface area. Negotiate AI-specific provisions: model training data handling, embedding security, inference log management, memorisation testing commitments, and model deletion protocols. This is typically the hardest phase because vendors resist these terms. Be prepared to walk away from vendors who won't commit.
Days 36–55: Privacy Impact Assessment
Conduct AI-specific privacy impact assessments for each system identified in the audit. Evaluate: minimum necessary compliance at the feature level, de-identification adequacy (including re-identification risk testing), memorisation exposure, embedding security posture, and inference log handling. Score each system against the PHI Surface Area Model.
Days 56–75: Technical Safeguards
Implement AI-specific security controls based on PIA findings. Priority order: (1) secure inference logs, (2) encrypt embedding stores, (3) implement memorisation probing on a monthly cadence, (4) deploy output monitoring for PHI leakage patterns, (5) establish adversarial testing protocols. Each control should map to a specific PHI surface.
Days 76–90: Monitoring & Governance
Establish continuous monitoring for PHI leakage in model outputs, not just at launch, but ongoing. Set up automated alerts for compliance drift as models are updated. Create a quarterly AI compliance review cadence. Document everything for OCR audit readiness. The output: a living compliance programme, not a one-time assessment.
The Regulatory Horizon
Three regulatory developments every healthcare AI leader should be tracking:
- OCR AI-specific enforcement: The Office for Civil Rights has already put AI on notice. Its Section 1557 nondiscrimination rule now explicitly covers AI-enabled clinical decision support tools, and the proposed HIPAA Security Rule update treats AI training data as regulated ePHI. The first high-profile enforcement action against an AI system will reshape the market overnight. The organisations with compliance infrastructure in place will be positioned as trusted partners. Everyone else will be scrambling.
- State-level AI healthcare regulations: Colorado's AI Act (SB 26-189, effective January 1, 2027) already requires transparency and a right to human review for consequential automated decisions, and California and several other states are building their own AI-specific healthcare rules that go beyond HIPAA. Multi-state compliance is becoming a genuine operational challenge for health systems and health-tech companies operating nationally.
- The EU AI Act's healthcare provisions: The EU AI Act classifies most healthcare AI as high-risk. That classification is settled, but the high-risk obligations themselves, conformity assessments, technical documentation, ongoing monitoring, are not enforceable yet: the Digital Omnibus deferred them to 2 December 2027 for standalone clinical AI (Annex III) and 2 August 2028 for AI embedded in regulated medical devices (Annex I). GDPR obligations for EU patient data are separate and apply today regardless of that deferral. If you serve EU patients or partner with EU health systems, use the runway to build the Annex III controls before the 2027 deadline, not after. For the broader GDPR intersection, see my AI and GDPR Compliance Guide.
The first major OCR enforcement action against an AI system will be the healthcare AI industry's 'Cambridge Analytica moment.' The organisations that have built compliance infrastructure now won't just survive it. They'll benefit from the trust premium it creates.
From Compliance to Competitive Advantage
Here's the reframe I give every healthcare CTO who treats HIPAA as a cost centre: compliance infrastructure IS trust infrastructure. The organisations that build rigorous AI compliance don't just avoid penalties. They win partnerships, accelerate clinical adoption, and create competitive moats that compliance-lagging competitors can't cross.
The Governance Playbook shows how to turn compliance into operational advantage. The AI Use Case Canvas helps you evaluate which healthcare AI investments are worth the compliance investment in the first place. And if you're building or buying healthcare AI and want an external assessment of your PHI surface area before OCR comes knocking, that's the advisory conversation.
Your action item: run a memorisation probe on your highest-risk AI system this week. If you don't know how, or if the results concern you, reach out. This is exactly the kind of assessment I do in advisory engagements. It's the single highest-ROI compliance action you can take today.
That health system in the Midwest? After the memorisation probe, we spent six weeks building the compliance infrastructure in this guide: PHI Surface Area mapping, BAA modernisation, inference log lockdown, continuous output monitoring. The clinical decision support tool is still in production. It still serves clinicians well. But now every response is monitored for PHI leakage, the model is probed for memorisation quarterly, and the compliance team can show OCR exactly where PHI flows and exactly how it's protected at every point.
They didn't remove the AI. They made it trustworthy. That's what compliance should do.
Get Weekly Thinking
Join 2,500+ AI leaders who start their week with original insights.
Senior AI strategist helping leaders make AI real across four continents. Forbes Technology Council member, IEEE Senior Member.
Ajay's views, from 15 years in the field. Not legal or compliance advice. See full disclaimers →
Published by AI Exponent LLC