Using AI with patient data: a practical checklist for agencies

The fastest way to turn an AI project into a compliance problem is to paste patient information into a tool nobody vetted. The good news is that using AI safely with protected health information (PHI) is very achievable. It just takes discipline up front.

Here's the checklist I use before any AI tool touches patient data. It's practical guidance, not legal advice, so involve your compliance officer and counsel.

Before you start

  • Is there a Business Associate Agreement? Any vendor that processes PHI on your behalf needs a BAA. Many cloud and AI providers offer BAAs, but often only for specific services and configurations. Confirm the exact service you'll use is covered.
  • Where does the data go? Know which region it's processed in, whether it's stored, and for how long.
  • Is it used for training? Get it in writing that your data won't be used to train the vendor's models.
  • Who on staff is using consumer tools? Free chat tools are rarely covered by a BAA. Set a clear policy and give staff an approved alternative.

When you design the workflow

  • Minimum necessary. Send the agent only the fields it needs. An eligibility check needs member ID and date of birth, not the visit notes.
  • Keep it in your tenant. Where possible, run AI services inside your own cloud subscription, behind your own identity and network controls.
  • Put a person in the loop. Decide which outputs need human approval. Anything clinical, coding-related or billing-related should.
  • Log everything. Record what went in, what came out, and who approved it. You'll need this for audits and for improving the system.

Once it's running

  • Measure accuracy. Spot-check outputs regularly against what a person would have done.
  • Review access. Include AI service accounts and keys in your regular access reviews.
  • Have an off switch. Every agent should be easy to pause without breaking the underlying workflow.