Privacy as the Real Case for Local Models

Cost is the argument people make for local inference and privacy is the argument that actually holds. But “privacy” is four different concerns wearing one word, and local deployment answers three of them completely and one of them not at all.

Start with the threat model

A threat model is three questions, and answering them takes less time than the argument you would otherwise have. What data is involved. Who must not see it. What happens if they do.

The point of writing it down is that it makes the requirement testable. “We are worried about privacy” cannot be satisfied. “Patient identifiers must not be transmitted to a third party under any circumstance, because the consent we obtained does not cover it” can be, and it tells you immediately that no contractual assurance from a vendor closes the gap. Different answers point at different solutions, and only some of them point at a GPU under your desk.

Four different worries

• Training on your data. The fear that prompts become future model weights. This is the most common worry and the most often already addressed — business and enterprise API tiers commonly commit contractually not to train on submitted data. If this is your only concern, read the terms before you buy hardware.
• Retention and logs. Even without training, requests may be stored for some window for abuse monitoring, and those stores are subpoenable and breachable. Retention periods vary by tier and some providers offer zero-retention arrangements. This is a contract question with a real answer.
• Transmission at all. Some obligations are not about what the recipient does but about whether the data crosses a boundary — a jurisdiction, a legal entity, an air gap, a consent scope. No contract satisfies a rule of this shape. This is the worry local inference exists for.
• Availability and continuity. Not privacy strictly, but it travels with it: a model on your disk cannot be deprecated, rate-limited, price-changed or withdrawn. For anything that must keep working unchanged for years, this is a serious argument on its own.

Workloads where local plainly wins

• Regulated personal data under a narrow consent. Health records, legal files, HR investigations, safeguarding material. Where lawful processing rests on a consent or a basis that does not extend to a new processor, the cheapest compliance answer is that there is no processor.
• Data residency and cross-border transfer. An obligation to keep data in one jurisdiction is satisfied cleanly by hardware in that jurisdiction, and messily by everything else.
• Air-gapped and field environments. Ships, substations, secure facilities, clinical devices. There is no network to send to, and the model on disk is the only option that exists.
• Source code and trade secrets under strict policy. Where the rule is categorical rather than risk-weighted, a local assistant is what turns a blanket prohibition into an allowed tool.
• Bulk processing of sensitive corpora. Classifying or embedding millions of internal documents concentrates the exposure of the whole archive into one pipeline — and happens to be exactly the high-utilisation shape where self-hosting is also cheap.
• Long-lived embedded systems that must behave identically in five years, with no vendor in the loop.

What local does not fix

This is the section that makes the rest of the page trustworthy. Running the model yourself moves the trust boundary; it does not remove the need for one.

• Access control is now entirely yours. A team server with an open port on the LAN and no authentication is a worse privacy posture than a contracted vendor. Local inference silently transfers the whole of the security job to you, and the job is larger than the download was.
• Logs still exist. Whatever you built around the model probably records prompts somewhere, and those records are on a disk that is backed up to somewhere. The transcript is the sensitive artefact regardless of where inference happened.
• Prompt injection is unaffected. If your local model reads untrusted documents and can call tools, it can be induced to exfiltrate — and now the tools it can reach are internal ones. Local deployment can make this class of problem worse, not better.
• The supply chain is real. You downloaded weights from somewhere. Verify hashes, prefer formats that cannot execute code, and know that a fine-tune can encode behaviour you did not audit and cannot easily detect.
• Compliance is a process, not a topology. Records of processing, retention schedules, subject-access handling and deletion still apply to data you process on your own hardware. Nothing about a local GPU removes an obligation.

The options between the extremes

Local and public API are the ends of a spectrum with several useful points in between, and picking the right one usually costs less than either extreme:

• Zero-retention or enterprise API tiers — closes the training and retention worries by contract, leaves transmission open.
• Single-tenant or in-region hosted deployment — dedicated capacity in a chosen jurisdiction, run by someone else.
• Your own cloud account — the model runs on infrastructure inside your existing compliance perimeter, which for many organisations is the boundary that the paperwork actually names.
• Redaction or tokenisation before the call — strip or pseudonymise identifiers locally, send the rest. Effective for structured identifiers, unreliable for free text where identity leaks through detail.
• Split routing — a local model for anything touching regulated data, a hosted model for everything else, chosen by a classifier you control.

Work down that list against your threat model and stop at the first option that satisfies it. Buying hardware to solve a problem that a contract clause already solved is a common and expensive mistake; so is signing a contract for a problem that only an air gap solves.