On-Premise AI for Regulated Industries

Self-hosting a model is usually proposed as the answer to a compliance problem and evaluated as a hardware purchase. Both framings are incomplete: it closes a specific and short list of questions, opens another list, and its true cost is dominated by a line item that never appears in the first version of the spreadsheet.

Engineering guidance, not legal advice. Whether self-hosting is necessary for your obligations is a question for counsel; whether it is affordable and whether it works are questions for you, and this page is about those.

It is a spectrum, not a switch

Most conversations that start at row one and jump to row five would have been better served by row three. Establish which question you are actually trying to close before choosing a row, because each step down costs meaningfully more and closes progressively fewer additional questions.

The cost model, line by line

Fill this in with your own quotes; no numbers are supplied here because any number printed on a page about accelerator pricing is wrong by the time it is read. The point is the completeness of the list, since the omissions are what make self-hosting look cheaper than it is:

ONE-OFF
  accelerators + spare capacity for failure
  host servers, high-speed interconnect, NICs
  rack, power distribution, cooling upgrade
  installation, network, security review
  model evaluation + selection effort

RECURRING — INFRASTRUCTURE
  power (compute + cooling), at your industrial rate
  data centre space, or cloud instance hours
  hardware support contract / replacement reserve
  depreciation over a realistic life (accelerator
    generations turn over faster than servers do)

RECURRING — PEOPLE            <-- the line that decides it
  inference platform engineering (serving stack,
    batching, quantisation, upgrades)
  24/7 on-call for a production dependency
  evaluation + regression testing per model change
  security patching of the whole stack

RECURRING — OPPORTUNITY
  latency of adopting a new model (weeks, not hours)
  capability gap vs the frontier, priced as
    engineering time spent compensating for it
  capacity headroom you pay for and do not use

UTILISATION — the multiplier
  effective tokens/sec achieved  vs  peak spec
  average utilisation across the day
  → cost per token = total annual cost
                     / (tokens/sec x utilisation x seconds)

Two things usually decide the comparison. Utilisation is the first: hardware bought for peak and idle at night can be several times more expensive per token than the same hardware at high steady load, and bursty interactive traffic is the worst possible shape for owned capacity. The people line is the second — a serving platform is a product with an on-call rotation, and pricing it at zero because the team already exists is the single most common error in these business cases.

Self-hosting wins on cost most convincingly at high, steady, predictable volume with a model that is good enough and stays good enough. That is a real workload shape — batch classification, embedding generation, document processing — and it is not the shape of most interactive products.

Questions it genuinely closes

• Third-party processing of content. If nothing leaves, there is no processor, no sub-processor chain, no transfer question and no retention clause to negotiate. This is the real prize and it is a large one.
• Training on your data. Unanswerable questions about somebody else’s pipeline become answerable questions about yours.
• Residency, definitively. You know where the hardware is because you can walk to it.
• Version stability. The model changes when you change it. For anything requiring validation and re-validation, this is worth more than it first appears.
• Availability of a discontinued model. A hosted model can be retired on the provider’s schedule; weights on your disk cannot.

Questions it moves onto you

None of these disappear; they change owner, and the new owner is your team. Access control to the inference infrastructure and to whatever logs it produces. Patching a stack that now includes a serving engine, drivers and an accelerator firmware surface. Evidence of all of it for the same audits you were preparing for anyway — self-hosting removes the vendor questionnaire and adds a control environment you must describe yourself.

And the licence. Open weights are not uniformly open: some carry field-of-use restrictions, usage thresholds, or conditions on derivative models. Read the licence for the specific model version you intend to deploy, and read it before the hardware order rather than after, because it is the cheapest possible moment to discover a restriction.

Deciding without a spreadsheet fight

Two questions, in order. First: which specific obligation requires this, in a sentence, with the source? If nobody can produce it, the driver is discomfort rather than requirement, and dedicated capacity or a well-chosen region usually resolves discomfort far more cheaply. Second: what fraction of our traffic carries the data that creates the obligation? It is very often a small fraction, and a split architecture — sensitive workloads on owned capacity, everything else on an API — beats a uniform decision in both directions.