Local Inference as Part of a HIPAA Compliance Strategy

Moving inference in-house removes one third party from the chain. It does not remove a single safeguard, and it creates several new copies of protected health information that you are now responsible for. The net effect on your compliance work is usually to increase it.

The one thing that genuinely changes

HIPAA requires a covered entity to obtain satisfactory assurances, through a written business associate agreement, before it discloses protected health information to a business associate — 45 CFR 164.502(e)(1) on the Privacy Rule side and 45 CFR 164.308(b) on the Security Rule side. The US Department of Health and Human Services publishes sample business associate agreement provisions setting out what those assurances look like.

If a prompt containing PHI never leaves hardware you control, there is no disclosure to an inference vendor for that step, and therefore no BAA needed for that step. That is a real result. It removes a vendor from your agreement chain, removes a subprocessor from your record of processing, and removes a party whose own breach would become your notification obligation. It also removes a negotiation: not every model provider will sign a BAA, and the ones that do often restrict which models are in scope. If your blocker is “procurement will not approve another data processor”, local inference genuinely solves that blocker. The general treatment of what a vendor BAA has to contain is in the clauses to look for in an AI vendor’s BAA.

That is the whole of what changes. It is one clause in one rule.

What is completely unchanged

The Security Rule applies to electronic PHI wherever a covered entity or business associate creates, receives, maintains or transmits it. A GPU under a desk in your own building is squarely inside that scope, and every standard applies to it exactly as it applies to a server rack.

• Administrative safeguards, 45 CFR 164.308. The risk analysis at 164.308(a)(1)(ii)(A) is a required implementation specification, and a new inference host is a new system to analyse. Sanction policy, workforce clearance, information access management and contingency planning all attach to it.
• Physical safeguards, 45 CFR 164.310. Facility access controls, workstation security, and — the one that bites — device and media controls at 164.310(d)(1), covering disposal and media re-use. The disk holding your model and its logs cannot simply be thrown away or resold.
• Technical safeguards, 45 CFR 164.312. Access control with unique user identification (required), emergency access (required), automatic logoff and encryption (addressable); audit controls; integrity; person or entity authentication; transmission security. HHS is explicit that “addressable” does not mean optional — you implement it, or document why it is not reasonable and appropriate and implement an equivalent alternative.
• Minimum necessary, 45 CFR 164.502(b). Still governs what you put in the prompt. Pasting a whole chart into a local model because the tokens are free is exactly the failure this standard exists to prevent; see minimum necessary applied to an AI assistant.
• Breach notification, 45 CFR 164.400–414. A stolen laptop with a local model and six months of chat history is an impermissible acquisition of PHI, assessed the same way as any other. Encryption at rest is what moves it out of “unsecured PHI”, and that depends on the encryption meeting HHS’s published guidance, not on the data having been local.

A default llama.cpp or Ollama install satisfies almost none of this. It has no user accounts, so 164.312(a)(2)(i) unique user identification has nothing to hook into. It has no authentication on its HTTP endpoint, so 164.312(d) is unmet. If it is bound to 0.0.0.0 it is answering unauthenticated requests from the whole subnet. None of that is a criticism of the software — it is not sold as a clinical system — but it means the compliance work is in front of you, not behind you.

The new ePHI local inference creates

This is the part that surprises people. Moving inference in-house converts a transient disclosure into durable local storage, and durable local storage of PHI is a bigger obligation than a transient disclosure under a BAA.

The artefacts are concrete: readline history from an interactive CLI; a prompt cache written to disk by a runtime flag such as llama.cpp ’s --prompt-cache; the chat database of whatever web front end you put in front of the server; the server’s own request logs; shell history if prompts are passed on a command line; swap and hibernation files holding whatever was in RAM. Each of those is now ePHI on a system you must safeguard, back up under your contingency plan, include in your risk analysis, and dispose of under 164.310(d)(2). The wider inventory of these is in what local inference cannot guarantee about privacy.

Backups deserve their own sentence. A local inference host that is swept up by the same nightly backup as everything else has just put PHI wherever those backups go — including, in a lot of small practices, a consumer cloud sync folder. That is a disclosure, and it is one the “runs locally” framing actively hides.

Audit controls are where local setups fail

45 CFR 164.312(b) requires mechanisms that record and examine activity in systems containing ePHI. This is the standard local inference stacks are furthest from meeting, and the gap is structural rather than a configuration mistake.

A hosted API has per-key identity: every request is attributable to a credential, and the provider produces logs. A bare local server has an HTTP port. It cannot tell you who asked, because nobody told it. To get to a defensible position you have to put something in front of it that authenticates a person, records the request, and can answer “which member of staff sent this patient’s data to the model, and when”. That is a piece of software somebody has to own and operate, and the honest framing is that local inference moved that job from a vendor to you rather than eliminating it.

There is no HIPAA certification

HHS does not certify, endorse or accredit anyone for HIPAA compliance, and no government-issued HIPAA certification exists. Compliance is a state you are in and can evidence, assessed against your own risk analysis; it is not a badge, and it is not conferred by an architecture.

Which means: “HIPAA-compliant because it runs locally” is not a claim anyone can make about a piece of software, including about software you write. A vendor advertising a “HIPAA-certified local AI appliance” is selling you their own attestation plus, at best, a third-party audit against a framework that is not HIPAA. Read what the audit actually covered.

The moment you fall back to a hosted model

Most real deployments are not purely local. A small model handles the routine work and something larger is called when it cannot cope, which is the pattern described in hybrid local and API inference. That fallback is a disclosure, and it needs a BAA with whoever receives it, exactly as if the local model were not there.

The consequence worth internalising is that the routing decision becomes a compliance control. A rule that says “overflow to the hosted model when the context is too long” is a rule about when PHI leaves the building, and it will fire on the longest, most detailed, most identifiable inputs — precisely the ones you least wanted to send. If that route exists it must be deliberate, logged, restricted to a provider under BAA, and testable. Configuring it as a performance optimisation and discovering later that it was a disclosure path is the failure mode to design against.