HIPAA-Compliant AI: What Healthcare Leaders Need to Know

No AI tool is HIPAA compliant on its own. Compliance comes from how a tool is contracted, configured, connected, and used inside your organization. A vendor can support compliance. Only your organization can achieve it.

That distinction matters because healthcare teams are under pressure to adopt AI quickly. Clinicians want help with documentation. Revenue cycle teams want help with denials. Patient access teams want help with scheduling and calls. Each of those use cases touches protected health information, and each one needs the right safeguards.

This guide is for CIOs, CISOs, compliance officers, and operations leaders at mid-market health systems, physician groups, and health tech companies. It explains what HIPAA compliant AI actually requires, how to vet vendors, how to design the architecture, and how to move from pilot to production without creating new risk. It is general guidance, not legal advice, so involve your privacy officer and counsel in every deployment.

When HIPAA applies to AI

HIPAA applies to covered entities, which include most health plans, healthcare clearinghouses, and providers that conduct certain transactions electronically. It also applies to their business associates: vendors that create, receive, maintain, or transmit protected health information on their behalf.

When a covered entity uses an AI tool that processes PHI, the vendor behind that tool is usually a business associate. That means the vendor must sign a business associate agreement, and both sides must meet their obligations under the Privacy, Security, and Breach Notification Rules.

HIPAA does not apply to every health-related AI use. A consumer wellness app that a person downloads on their own may fall outside HIPAA, though other privacy laws and the FTC may still apply. Data that has been properly de-identified under HIPAA is not PHI. But most AI projects inside a health system, physician group, or health plan will be inside HIPAA’s scope.

The core requirements for HIPAA compliant AI

A business associate agreement that fits the AI workflow

Every vendor in the chain that touches PHI needs a BAA. That includes the AI application vendor, the model provider if PHI reaches the model, the cloud host, and any subcontractors. Legal analysis from firms such as Morgan Lewis has stressed that the BAA should explicitly permit the data flows the AI system actually uses, and that covered entities should look closely at vendor permissions for de-identification and data aggregation, since many AI vendors include those rights in their standard terms.

Read the BAA alongside the vendor’s main contract and data processing terms. Look for clear limits on how the vendor can use your data, including whether it can use PHI to improve or train its models.

Minimum necessary

HIPAA’s minimum necessary standard means using and disclosing only the PHI needed for the task. For AI, that is a design requirement. A scheduling assistant does not need the full medical record. A coding assistant may need clinical notes but not the patient’s address. Build each use case so the AI sees only what it needs.

Security Rule safeguards

The HIPAA Security Rule requires administrative, physical, and technical safeguards for electronic PHI. For AI, that includes a risk analysis that covers the new system, access controls tied to roles, encryption in transit and at rest, audit logging of who accessed what, and integrity controls. The Security Rule also requires periodic evaluation, so your AI systems need to be part of your regular security review, not a one-time approval.

Training data rules

The most important question for many AI vendors is whether your PHI will be used to train their models. Guidance from healthcare law firms is consistent: PHI should not go into public AI tools or be used to train general-purpose models. Training or fine-tuning on PHI is more defensible only in a closed environment built by or for your organization, and only after a careful analysis of HIPAA permissions. If a vendor cannot give a clear, contractual answer on training, do not send them PHI.

De-identification done properly

HIPAA recognizes two methods of de-identification: Safe Harbor, which removes eighteen specified identifiers, and Expert Determination, in which a qualified expert concludes the risk of re-identification is very small. De-identified data can be very useful for analytics and model development. But re-identification risk grows when large datasets are combined, so treat de-identification as a formal process, not a quick cleanup step.

Breach readiness

AI systems create new places where PHI can leak: prompts, logs, cached outputs, and vector databases used for search. Map these locations, include them in your incident response plan, and make sure your logs do not quietly become a second copy of the medical record.

How to vet an AI vendor for HIPAA

Many vendors describe their products as HIPAA compliant. Treat that phrase as the start of a conversation, not an answer. Ask these questions.

Will you sign a BAA, and does it cover every subcontractor and model provider in the data flow?

Will our data, prompts, or outputs be used to train or improve any model? Where is that stated in the contract?

Where is PHI processed and stored, and for how long? Can retention be set to zero for prompts?

What access controls, encryption, and audit logs do you provide, and can we export the logs?

Do you have an independent security assessment, such as SOC 2 Type II or HITRUST, and does it cover the AI features?

How do you handle de-identification, and who performs it?

How do you notify us of a breach, and within what time frame?

What happens to our data when the contract ends?

A vendor with strong answers will respond quickly and in writing. A vendor that hesitates is telling you something important.

Architecture patterns for HIPAA compliant AI

Keep PHI inside your environment where possible

The safest pattern is to keep PHI inside infrastructure you control, such as your own cloud tenant with a BAA, and send only what is needed to the AI model. Many enterprise model providers now offer deployments inside major cloud platforms under a BAA, which makes this pattern practical for mid-market organizations.

Use an orchestration layer as the gatekeeper

Put a layer between your clinical and business systems and the AI model. This layer enforces minimum necessary, strips or masks identifiers when the use case allows it, logs every request and response, and applies role-based permissions. It also lets you change models later without redesigning your controls. Our guide on model-agnostic AI and avoiding vendor lock-in explains why that separation is worth building in from the start.

Ground answers in approved sources

For assistants that answer questions, connect the AI to approved documents and records rather than letting it rely on general knowledge. This reduces wrong answers and makes it easier to show where an answer came from.

Keep a human in the loop for clinical and coverage decisions

AI can draft, summarize, and suggest. For anything that affects diagnosis, treatment, or coverage, a qualified person should review and decide. Build that review step into the workflow and log it.

Healthcare AI use cases and their HIPAA considerations

Ambient clinical documentation records and summarizes visits. It touches highly sensitive PHI, so it needs a strong BAA, clear patient notice practices, and clinician review of every note.

Revenue cycle automation, such as prior authorization support, coding assistance, and denial management, touches claims and clinical data. It benefits from strict minimum necessary design and good audit logs.

Patient access tools, such as scheduling assistants and call center support, touch demographics and appointment data. They need clear identity verification and a fast path to a person.

Operational analytics, such as capacity planning and no-show prediction, can often use de-identified or limited data sets, which lowers risk.

Each of these can be done in a compliant way. None of them is compliant by default.

A deployment checklist

Define the use case, the PHI it needs, and the people who will use it.

Confirm HIPAA applies and identify every vendor in the data flow.

Sign BAAs that cover the actual AI workflow, including subcontractors.

Confirm in writing that PHI will not train shared models.

Run a risk analysis that covers the new system, its logs, and its storage.

Design for minimum necessary, with masking or de-identification where possible.

Set role-based access and turn on audit logging.

Build human review into any clinical or coverage decision.

Train staff on approved use and on what not to put into AI tools.

Add the system to periodic security evaluations and incident response plans.

Governance that does not slow you down

A short, clear AI policy beats a long one nobody reads. Name the approved tools, the data each one can handle, and who to ask when something is unclear. Create a small review group with compliance, security, clinical, and IT leaders that can approve new use cases in weeks, not months. Track every AI system in an inventory with its owner, data, vendors, and review date.

Staff will use AI with or without a policy. Giving them approved, safe tools is the best defense against shadow AI, where people paste PHI into consumer tools because nothing better is available.

Common HIPAA mistakes with AI

The first mistake is assuming a vendor’s marketing claim settles the question. A vendor that says its product is HIPAA compliant may only mean that it will sign a BAA for certain plans, or that its infrastructure has security certifications. Always confirm the specific product, plan, and configuration you are buying are covered.

The second is letting staff use consumer AI tools because nothing approved is available. Clinicians and administrators who are short on time will paste notes into whatever tool helps them. The fix is to provide approved tools quickly, not to rely on policy alone.

The third is forgetting the logs. Prompts and responses often contain PHI, and many systems log them by default. If those logs sit in a monitoring tool without a BAA or with broad access, you have created a new exposure.

The fourth is sending the full record when a small slice would do. Pulling an entire chart into a prompt to answer a scheduling question violates the spirit of minimum necessary and increases risk for no benefit.

The fifth is treating approval as permanent. Models, vendors, and features change. A tool that was reviewed last year may now send data to a new subprocessor or offer a new feature that changes the risk. Review AI systems on a schedule, the same way you review other critical systems.

Frequently asked questions

Is there such a thing as HIPAA certified AI?

No. HHS does not certify products as HIPAA compliant. Vendors can support compliance through contracts, controls, and independent assessments, but compliance depends on how your organization uses the tool.

Can we use a general AI chatbot with patient data?

Only if the vendor signs a BAA covering that use and your configuration meets HIPAA requirements. Consumer versions of AI tools generally should not be used with PHI.

Does de-identified data fall under HIPAA?

Properly de-identified data, using Safe Harbor or Expert Determination, is not PHI. Be careful, because combining datasets can raise re-identification risk.

Who is responsible if an AI vendor has a breach?

Both parties have obligations. The business associate must protect PHI and report breaches, and the covered entity must manage its vendors and respond. Your BAA should spell out timelines and responsibilities.

How long does a compliant AI deployment take?

A focused use case with a vendor that already supports BAAs can move from review to pilot in a few months. Most of the time goes to contracts, risk analysis, and integration, not the AI itself.

How Xcelacore AI helps healthcare organizations

Healthcare is one of Xcelacore AI’s priority industries, and our clients have included Northwestern Medicine. We help mid-market healthcare organizations plan AI use cases, design architectures that keep PHI protected, build and integrate the tools, and train teams to use them well.

If you are comparing partners, see our guides to the best healthcare AI consultants and developers and the leading HIPAA-compliant software development companies.

Questions?

—

We’re happy to discuss your technology challenges and ideas.