Multi-Region AI Serving

Multi-region is three different projects wearing one name: reducing latency, satisfying data residency, and surviving a regional failure. They need different architectures, and conflating them produces a system that is expensive at all three. The latency case in particular is weaker for AI services than for web applications, and the arithmetic below shows why.

Three different reasons, three architectures

Write down which one you are solving before choosing anything, because the answers conflict. Residency forbids the replication that availability wants. Latency wants traffic to move freely between regions; residency forbids exactly that. A system that must satisfy both ends up as several independent single-region stacks with a shared control plane, which is a legitimate design and quite different from what people picture when they say multi-region.

The latency floor, derived

There is a lower bound on network latency set by physics, and it is worth computing because it tells you how much of your latency budget geography can possibly buy back.

Light in a vacuum:            c   = 299,792 km/s
Light in optical fibre:       ~0.67c ≈ 200,000 km/s
  (the refractive index of silica; this is the number to use)

One-way propagation delay = distance ÷ 200,000 km/s
Round trip = 2 × that, over the path the fibre actually takes.

Real routes are not great circles. A commonly used engineering
approximation is to multiply great-circle distance by about 1.4 to allow for
routing, which is a rule of thumb, not a law — measure your own path.

Worked, London to New York:
  great-circle distance ................ 5,570 km
  × 1.4 route factor ................... 7,800 km of fibre
  one way = 7,800 / 200,000 ............ 39 ms
  round trip ........................... 78 ms

Worked, London to Sydney:
  great-circle ......................... 16,990 km
  × 1.4 ................................ 23,800 km
  round trip ........................... 238 ms

Then add, per connection, on top of propagation:
  TCP handshake ........................ 1 round trip
  TLS 1.3 handshake .................... 1 round trip (2 for TLS 1.2)
  so a cold HTTPS connection to Sydney costs roughly 3 × 238 = 714 ms
  before a single byte of your request is processed.

The handshake multiplier is why connection reuse and keep-alive matter more than region placement for many workloads: eliminating two round trips on a long path saves more than moving the endpoint a thousand kilometres closer.

Why the latency win is small here

Now put the network number next to the rest of an AI request. Take a user in Sydney and a service in London, generating a 400-token answer:

  network round trip (warm connection) ............  238 ms
  time to first token (prefill + queue) ...........  600 ms
  generation, 400 tokens at 40 tokens/s ........... 10,000 ms
  -----------------------------------------------------------
  total ........................................... 10,838 ms

Move the service to Sydney and eliminate the network entirely:
  saved 238 ms out of 10,838 ms = 2.2%

Move it, and the perceived first-token latency improves from
838 ms to 600 ms — a 28% improvement on the number the user
actually notices first.

Both readings are true and they point in different directions. Total time is dominated by generation, so regional placement barely moves it. Time to first token — the number that decides whether an interface feels responsive — improves substantially, and time to first token versus tokens per second is about why those two are separate metrics.

The practical conclusion: do not build multi-region for the latency of long generations. Do consider it for short, chatty, latency-sensitive interactions — autocomplete, classification, voice — where the network is a large fraction of the total. And before either, check the cheaper fixes: streaming converts the total into the first-token number for the user; semantic caching and prompt caching remove the request entirely. Each is a fraction of the cost of a second region.

Classifying your state

This is the section that determines the architecture. Take every piece of state and put it in one of three buckets. The exercise takes an afternoon and it is the deliverable.

Two entries in that last row surprise people. Embeddings are derived from the source text and are not anonymisation — they are a lossy but often invertible-enough representation, and regulators have not treated them as a laundering step. And logs are the most common leak: the application respects the boundary and the observability pipeline ships full request bodies to a collector in another jurisdiction. PII in LLM logs is about exactly that, and EU data residency for AI covers what the obligations actually say.

The model provider is part of this boundary too. If your regional stack calls an inference endpoint that processes in another region, the boundary is broken at that hop no matter how carefully your own storage is arranged. Ask providers where inference physically happens, which region options exist, and what they retain — zero data retention covers the terms to look for.

Routing requests to a region

Three mechanisms, and they are not interchangeable.

• Latency- or geo-based DNS. Resolvers get an answer appropriate to their location. Simple and universally supported. Failover speed is bounded by TTL and by resolvers that ignore TTL, so treat it as minutes, not seconds.
• Anycast. One address announced from many locations; the network picks. Fast failover and no client involvement, but you do not control the choice, which makes it a poor fit when the requirement is residency rather than latency.
• Application-level routing. A thin edge layer reads the tenant identifier and forwards to that tenant’s home region. Slower by one hop, and the only one of the three that can express “this customer’s data lives in region X” correctly. For residency, this is the answer.

Whichever you pick, pin the tenant. A request that lands in the wrong region should be forwarded there or refused, never served locally with a copy of the data pulled across. The forwarding hop costs a round trip and the refusal costs a support ticket; a silent copy costs a notification to a regulator.

Failover across a residency boundary

Here is the conflict in its sharpest form: region A is down, region B has capacity, and the data may not go to B. There are only four honest options, and choosing between them is a business decision that must be made before the incident, not during it.

1. Fail to another region inside the same jurisdiction. The clean answer where one exists. It is the reason to check, at design time, how many compliant regions your provider offers — if the answer is one, you have no in-jurisdiction failover and you should know that now.
2. Degrade in place. Serve what can be served without the unavailable component: cached answers, a smaller local model, read-only mode, a queue that accepts work for later. Graceful degradation is the design pattern.
3. Fail closed. Return an honest error. For some regulated workloads this is the only permitted answer, and saying so explicitly in the runbook is better than an on-call engineer improvising a cross-border failover at 3am.
4. Fail over with consent. Some contracts permit cross-border processing in a declared emergency with notification. If yours does, the notification template and the approver’s name belong in the runbook, not in a lawyer’s inbox during the outage.

Whatever you choose, it goes in the runbook as a written decision with a name attached. On-call runbooks has the format, and disaster recovery covers what has to be recoverable in the surviving region for any of these to be possible.

The smallest thing that works

Most teams that need multi-region need less of it than they build. In rough order of cost, the ladder is:

1. One region, plus edge caching and streaming. Removes most of the perceived latency for most users, at almost no architectural cost.
2. One region, plus regional API entry points. A thin proxy near the user terminates TLS and holds a warm pooled connection back to the origin. This removes the handshake round trips, which the arithmetic above showed are the larger share of the network cost. No data is stored at the edge.
3. Second region as a warm standby. Deployed, weights cached, receiving no traffic except health checks and a small continuous share to prove it works. Solves availability without solving latency or residency.
4. Full per-jurisdiction stacks. Independent deployments sharing only code and configuration. Solves residency, and it is the most expensive thing here — build it when a contract requires it, not in anticipation.

The warm standby deserves the note about the small continuous share of traffic. A standby that receives none is a standby whose certificate expired, whose weights are a version behind and whose IAM policy was changed six months ago. Send it one per cent of real traffic and the failover you are relying on is a thing you observe daily rather than a thing you believe in.