Putting Azure API Management in Front of Azure OpenAI

An Azure OpenAI resource has one rate limit per deployment and two API keys. Neither of those is a way to tell one team from another, and that gap is the entire reason API Management ends up in this position.

What APIM adds that the resource cannot

Three things, and it is worth being precise about them because APIM is expensive to run for the ones it does not add.

• Identity per caller. An APIM subscription key identifies a consumer. The Azure OpenAI resource key identifies nobody — every team using it looks identical in every metric.
• Token-aware limits. APIM ships policies that count prompt and completion tokens rather than requests, so a per-team budget can be expressed in the unit the bill is denominated in.
• Rotation without redeployment. The resource key lives once, in APIM. Rotating it is one change; consumers keep their own unchanged subscription keys.

What it does not add is capacity. Everything downstream still shares the deployment’s TPM allocation, and a policy that limits one team does not create headroom for another — it only decides who gets the 429 first.

The backend and the credential

Import the Azure OpenAI resource as an API and define it as a backend object rather than a raw URL, because backends are what circuit breakers and load-balanced pools attach to later. The inbound policy then supplies the credential.

Use the APIM instance’s managed identity rather than a key. Microsoft’s Azure OpenAI RBAC documentation is clear that the role permitting inference calls with Microsoft Entra ID is Cognitive Services OpenAI User, and that Cognitive Services Contributor — despite being the more powerful management role — cannot make inference calls with Entra ID at all. Assign the former to the APIM identity on the OpenAI resource:

<inbound>
  <base />
  <authentication-managed-identity
      resource="https://cognitiveservices.azure.com"
      output-token-variable-name="aoai-token" />
  <set-header name="Authorization" exists-action="override">
    <value>@("Bearer " + (string)context.Variables["aoai-token"])</value>
  </set-header>
  <set-backend-service backend-id="aoai-weu" />
</inbound>

With that in place there is no Azure OpenAI key anywhere in the request path, which makes the rotation problem disappear rather than get centralised.

Per-team token quota

The policy that does the real work is documented as llm-token-limit — it was introduced as azure-openai-token-limit and generalised when APIM added support for the Anthropic Messages and Google Vertex AI schemas alongside OpenAI Chat Completions and Responses. Microsoft’s reference, dated 2026-04-01 at the time of writing, lists it as available on Developer, Basic, Basic v2, Standard, Standard v2, Premium and Premium v2.

It takes either a rate (tokens-per-minute), a quota (token-quota over a token-quota-period of Hourly, Daily, Weekly, Monthly or Yearly), or both. Exceeding the rate returns 429; exceeding the quota returns 403, which is a genuinely different signal and worth handling separately in clients.

<inbound>
  <base />
  <llm-token-limit
      counter-key="@(context.Subscription.Id)"
      tokens-per-minute="20000"
      token-quota="50000000"
      token-quota-period="Monthly"
      estimate-prompt-tokens="false"
      remaining-tokens-header-name="x-team-remaining-tokens"
      tokens-consumed-header-name="x-team-tokens-consumed" />
</inbound>

Keying on context.Subscription.Id gives one counter per APIM subscription, which is what makes this per-team. Microsoft’s note on counters is important if you apply the policy at more than one scope: a single counter is used for each distinct counter-key value across all scopes where the policy appears, so separate counters require separate key expressions.

Pair it with llm-emit-token-metric to write token counts into Application Insights dimensioned by the same key. That is how per-team attribution becomes a chart rather than a monthly export.

Be honest about what the layer costs before adding it. Every request now crosses an extra hop, which shows up in time-to-first-token on a latency-sensitive path; the counters are held per gateway, so a multi-region APIM deployment enforces the limit once per region rather than globally; and APIM itself is a resource with a monthly bill that is unrelated to how many tokens flow through it. For a single team calling one deployment, deployment-level quota already does the job and APIM is expensive scaffolding around a solved problem. It earns its place at the point where “which team spent this” becomes a question somebody actually asks.

The estimation trade-off

estimate-prompt-tokens is required and there is no safe default, because the two settings fail in opposite directions.

With false, the policy uses the actual token counts from the usage section of the model response. Accurate, but Microsoft documents that prompts may still be sent to the backend after the limit is exceeded — the overage is detected from the response, and only subsequent requests are blocked. You will pay for the request that crossed the line.

With true, prompt tokens are estimated from the API schema before the request goes out, so a request over the limit never reaches the model. Microsoft notes this may reduce performance, and that image inputs are overcounted as up to 1,200 tokens each.

Streaming removes the choice. Microsoft states that when stream: true is set, prompt tokens are always estimated regardless of the setting, and completion tokens are estimated too. Concurrency loosens it further: because the exact consumption is unknown until the response returns, near-simultaneous requests can temporarily exceed the configured limit. Treat these as budgets, not as hard ceilings.

Backend pools and circuit breakers

The second reason APIM appears in this position is failover across two Azure OpenAI resources. A backend pool supports round-robin, weighted, priority-based and session-aware balancing, and Microsoft documents that lower-priority groups are used only when every backend in a higher group is unavailable because a circuit breaker rule has tripped. That maps cleanly onto a provisioned deployment at priority 1 with a standard deployment at priority 2.

The circuit breaker is what makes it work, and it has one behaviour specific to this backend. Microsoft warns that an Azure OpenAI resource returning 429 can include a Retry-After header with a very large value — the documentation gives one day as an example — and that the circuit breaker’s dynamic trip duration will apply it. A rule that honours the header without a cap can take a healthy backend out of rotation for far longer than intended.