Skip to content
Multigrid

Infrastructure and operations

Running AI systems that are already live: containers, Kubernetes, autoscaling, spot capacity, deploys, monitoring, runbooks and the quarterly capacity review.

There is a large and good literature on how inference works and what a GPU does. There is much less on the hour after a model deploy goes wrong. These twenty pages are about the second thing: the manifests, the signals, the drain handlers, the rotation windows and the runbooks that decide whether an AI service degrades gracefully or falls over.

Everything here assumes something is already serving traffic. If you are still deciding whether to host a model at all, the arguments are in self-host versus API and rent versus buy; the mechanics of the inference server itself are in the inference cluster. This cluster starts where those end.

Two rules apply throughout. Where a number can be derived, it is derived on the page with its assumptions labelled, so you can substitute yours. Where a number belongs to a cloud provider — a quota, a notice period, a price — the page describes the mechanism and tells you to check the current documentation, because that is the honest state of the world and a confidently wrong ceiling costs more than a missing one.

Containerising an AI Service Properly

A multi-stage Dockerfile for a model-serving service, the CUDA base image decision, and how to work out where every gigabyte in your image came from.

10 min read

Kubernetes for Model Serving

The Deployment manifest for a GPU inference service: why GPU requests behave unlike CPU, the three probes a slow-loading model needs, and draining without cutting streams.

11 min read

Autoscaling on GPU: Metrics That Actually Work

Why GPU utilisation cannot drive an autoscaler for inference, why queue depth can, and the HPA configuration that follows from Little's law.

11 min read

Spot and Preemptible GPUs Without Losing Work

Checkpointing, drain handlers and the arithmetic that turns a headline spot discount into the saving you actually get once interruptions are priced in.

12 min read

Model Weights in CI/CD

Where a 40 GB artefact should live, how to reference it immutably, and how to get it onto the node and into page cache before the first request arrives.

11 min read

Canary and Blue-Green Deploys for Model Changes

Traffic splitting for model versions, an eval gate as the promotion criterion, and the sample-size arithmetic that says how long a canary must run to mean anything.

11 min read

Infrastructure as Code for AI Stacks

Module structure for GPU pools, queues and secrets, the lifecycle blocks that stop an autoscaler fighting your state file, and how to detect drift in CI.

11 min read

Rotating a Provider Key Without Downtime

The dual-key window, step by step: how to know when the old key has stopped being used, and how a compromised key changes the order of operations.

9 min read

Multi-Region AI Serving

What multi-region buys for an AI service, why the latency win is smaller than for a web app, and which state cannot legally cross a border.

12 min read

Priority Queues: Free Users Wait, Paying Users Do Not

Tiered queueing for an inference service: admission control, per-tenant concurrency caps, and the ageing rule that stops the free tier starving.

11 min read

Load Testing an AI Endpoint

A harness that models think-time and streaming rather than raw request rate, and why open-loop and closed-loop tests of the same service give different answers.

12 min read

Chaos Testing a Provider Outage

Fault injection at the client for model provider failures — the seven fault modes worth simulating, and the assertions that make each test mean something.

11 min read

Monitoring GPUs in Production

Which GPU counters mean something, why utilisation is not one of them, and the three alerts worth waking somebody for.

11 min read

Cost Allocation Across Teams and Environments

Tags on infrastructure, identifiers on requests, and how to join them into a chargeback report that teams accept rather than dispute.

11 min read

On-Call Runbooks for AI Services

Five runbooks for the five things that actually page you on an AI service, each with the first command, the branch, and what to do at each branch.

13 min read

Air-Gapped and Offline Deployments

Getting weights, dependencies and updates across an air gap, getting telemetry back out, and the parts of a normal AI stack that simply do not work offline.

11 min read

Serverless GPU: What It Is Good For

The break-even between paying for idle GPU hours and paying for cold starts, derived, and the four workload shapes on either side of it.

10 min read

Building an Internal Model Gateway

The architecture of a model gateway — routing, quota, audit, failover and key custody — and an honest accounting of what building one costs to run.

14 min read

Disaster Recovery for AI Systems

RTO and RPO applied to the assets an AI system actually holds — prompts, indexes, fine-tuned weights, conversation history and provider credentials.

11 min read

The Quarterly Capacity Review

Eight numbers to bring to the meeting, where each comes from, and the specific decision each one drives.

11 min read

Other topics