Setting Up Failover Between Two Azure OpenAI Deployments

Failover here means a second Azure OpenAI resource, in a second region, with its own endpoint. Everything else — a second deployment on one resource, a retry against the same endpoint — is a retry wearing a costume, and shares the failure it was meant to survive.

Two deployments that are actually independent

Create two Azure OpenAI resources in different regions and deploy the model to each. Three choices during creation decide whether this works.

1. Give the two deployments the same name. On Azure the model parameter of a request is your deployment name, not the model name. Identical names mean the failover swaps a base URL and a credential and changes nothing else about the request body.

   az cognitiveservices account deployment create \
     --resource-group rg-model --name aoai-weu \
     --deployment-name chat-prod \
     --model-name gpt-4o --model-version 2024-08-06 \
     --model-format OpenAI --sku-name GlobalStandard --sku-capacity 100
   
   az cognitiveservices account deployment create \
     --resource-group rg-model --name aoai-neu \
     --deployment-name chat-prod \
     --model-name gpt-4o --model-version 2024-08-06 \
     --model-format OpenAI --sku-name GlobalStandard --sku-capacity 100

2. Pin the model version on both. Regions do not necessarily carry the same versions, and an auto-updating deployment can drift so that your fallback answers differently from your primary. Discovering that during an incident is the worst time to discover it.
3. Check whether the two share a quota pool. This is the step that makes the difference between resilience and theatre. Microsoft documents Global Standard deployments of the same model and version as sharing one quota pool across all regions in a subscription. If both of the deployments above are GlobalStandard, they draw on the same allowance, and a failover triggered by a quota 429 arrives at a deployment with exactly the same problem. For quota resilience, the second deployment should be a different type — a DataZoneStandard or a provisioned deployment — or live in a different subscription. For outage resilience, two Global Standard resources in two regions are fine, because they are two entry points. Know which of the two you are buying. See how Global Standard pools quota.

Prefer Microsoft Entra authentication over keys here. Keys are per-resource, so a key-based failover carries two secrets, two rotation schedules and two chances to leave one stale. A managed identity with Cognitive Services OpenAI User on both resources uses one token audience and one code path.

The client

Two clients, an ordered list, and a breaker so a dead primary is not re-probed on every request:

import time, random, logging
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from openai import AzureOpenAI, RateLimitError, APIStatusError, APIConnectionError

token_provider = get_bearer_token_provider(
    DefaultAzureCredential(), "https://cognitiveservices.azure.com/.default"
)

def make_client(endpoint):
    return AzureOpenAI(
        azure_endpoint=endpoint,
        azure_ad_token_provider=token_provider,
        api_version="2024-10-21",
        max_retries=0,        # we own the retry policy here
        timeout=60.0,
    )

TARGETS = [
    {"name": "weu", "client": make_client("https://aoai-weu.openai.azure.com/")},
    {"name": "neu", "client": make_client("https://aoai-neu.openai.azure.com/")},
]

OPEN_UNTIL = {}               # target name -> monotonic time the breaker reopens
BREAKER_SECONDS = 30

def complete(**kwargs):
    now = time.monotonic()
    last_error = None

    for target in TARGETS:
        if OPEN_UNTIL.get(target["name"], 0) > now:
            continue                            # breaker open, skip this target

        try:
            return target["client"].chat.completions.create(
                model="chat-prod", **kwargs
            )
        except RateLimitError as exc:
            wait_ms = exc.response.headers.get("retry-after-ms")
            logging.warning("429 on %s, retry-after-ms=%s", target["name"], wait_ms)
            OPEN_UNTIL[target["name"]] = now + BREAKER_SECONDS
            last_error = exc
        except (APIConnectionError,) as exc:
            OPEN_UNTIL[target["name"]] = now + BREAKER_SECONDS
            last_error = exc
        except APIStatusError as exc:
            if exc.status_code >= 500:
                OPEN_UNTIL[target["name"]] = now + BREAKER_SECONDS
                last_error = exc
            else:
                raise                           # 400/401/403/404 are ours to fix

    time.sleep(random.uniform(0.5, 1.5))        # jitter before surfacing failure
    raise last_error

The breaker is the part people leave out and then regret. Without it, every request pays the primary’s full timeout before trying the secondary, so a regional outage does not degrade your latency, it multiplies it.

What to fail over on

• 429 — but read the headers first. Microsoft documents four distinct causes, and comparing x-ratelimit-limit-tokens against your configured TPM distinguishes a temporary protective reduction from a real allocation shortfall. Both justify moving traffic; only one justifies a quota request. See the 429 page.
• 500, 502, 503, 504 and connection errors — the region is unwell. Fail over immediately.
• Not 400, 401, 403 or 404. A malformed request, a missing role assignment or a wrong deployment name will fail identically on the secondary. Retrying it there doubles the latency of a failure and, if the request was partially processed, can double the cost too. Raise those.
• Content filter rejections are not failures. A response blocked by the content filter is the system working. Failing over is an attempt to launder it through another region, it will not succeed, and it should not.

Streaming breaks the simple version

The function above works because the whole response arrives before you decide anything. Streaming does not have that property: the connection can fail after you have already forwarded two hundred tokens to the user.

Three honest options, in ascending order of effort:

• Fail over only before the first chunk. Once one token has reached the client, surface the error. Simple, correct, and covers the common case where the failure is at connection time.
• Buffer a short prefix. Hold the first N tokens before forwarding anything. You can retry silently within that window at the cost of adding its duration to time-to-first-token — which is the one number streaming existed to improve.
• Restart and reconcile. Re-issue on the secondary and have the client replace what it has rendered. Only worth it for a UI that can express a correction; for an API caller it is worse than a clean error.

Whichever you pick, a mid-stream failure has already cost you the tokens generated before it. Count that in the retry budget rather than assuming a failed stream was free.

What this does not buy you

Say it plainly so nobody over-claims in a design review. This pattern survives one region’s Azure OpenAI being unavailable, and one deployment’s allocation being exhausted when the fallback is genuinely independent. It does not survive: an Entra outage that stops both token acquisitions, a subscription-wide quota exhaustion when both deployments share a pool, a model behaving differently across regions if you did not pin the version, or a bug in your own request construction.

It also has an operational cost that is easy to forget. Two resources means two sets of metrics, two sets of rate-limit headers, two diagnostic settings and two things to keep in Terraform. If you never exercise the secondary, you will find out during the incident that its deployment was created at a lower capacity, or that its role assignment was never made. Send a small share of steady-state traffic to it, or accept that it is untested.

At more than a couple of services, this logic wants to live in front of the applications rather than inside each of them. Azure API Management is the Azure-native way to do that — APIM in front of Azure OpenAI — and it moves the retry policy, the backend pool and the circuit breaker into policy configuration.