Managing Azure OpenAI Deployments With Terraform

Azure OpenAI is not one resource. It is an account that holds an endpoint and keys, and a set of deployments that each bind one model version to one quota allocation. Getting the split right is most of the work, because the deployment is the thing your application code names.

Two resources, not one

In the AzureRM provider, an Azure OpenAI service is an azurerm_cognitive_account with kind = "OpenAI", and every model you can call is a separate azurerm_cognitive_deployment attached to it. This is not bureaucratic. The distinction matters at request time: your client sends the account’s endpoint and key, but the deployment name in the URL path, and that name is yours to choose. The model identifier appears nowhere in the request.

The practical consequence is that a deployment name is an indirection you control. Name it chat and you can repoint it at a different model version by changing Terraform, with no application change. Name it gpt-4o-2024-11-20 and you have hard-coded a model version into every call site in your codebase.

The account

resource "azurerm_cognitive_account" "openai" {
  name                  = "acme-openai-weu"
  location              = "westeurope"
  resource_group_name   = azurerm_resource_group.ai.name
  kind                  = "OpenAI"
  sku_name              = "S0"
  custom_subdomain_name = "acme-openai-weu"

  # Force traffic through Private Endpoint / firewall rules only.
  public_network_access_enabled = false

  identity {
    type = "SystemAssigned"
  }
}

custom_subdomain_name is the argument that catches people. Azure AD (Entra ID) token authentication against a Cognitive Services account requires a custom subdomain; without one you are restricted to key authentication. Since keys are the thing you then have to store, rotate and scope, setting the subdomain at creation is what makes keyless auth possible later. It cannot be added by editing the resource in place without a replacement, so decide now.

The system-assigned identity is what a customer-managed key or an on-your-data integration will need. Enabling it costs nothing and adding it later is a separate apply.

The deployment

resource "azurerm_cognitive_deployment" "chat" {
  name                 = "chat"
  cognitive_account_id = azurerm_cognitive_account.openai.id

  model {
    format  = "OpenAI"
    name    = "gpt-4o"
    version = "2024-11-20"
  }

  sku {
    name     = "GlobalStandard"
    capacity = 30
  }
}

The model block takes a format as well as a name, and format is not always "OpenAI" — models from other publishers in the Azure AI catalogue use their own format value, and passing the wrong one is a documented source of confusion in the provider’s issue tracker. If a deployment fails with a model-not -found style error and the name is right, check the format.

The sku.name values documented for this resource include Standard, GlobalStandard, GlobalBatch, DataZoneStandard, DataZoneBatch, ProvisionedManaged, GlobalProvisionedManaged and DataZoneProvisionedManaged. The choice is a data-residency and capacity decision, not a performance one: the Global variants let Microsoft serve your request from any region in exchange for better availability, while DataZone constrains that to a geography. If you have a residency commitment to a customer, this field is the one that keeps it.

What capacity actually buys you

capacity is the field most likely to be set to a number somebody copied. The AzureRM provider documents it as tokens-per-minute with a unit of measure in thousands, and a default of 1 — meaning a deployment created without an explicit capacity is limited to roughly one thousand tokens per minute. That is not a working deployment for anything; it is a deployment that returns 429 on its second request.

So capacity = 30 is approximately 30,000 tokens per minute for that deployment. Two consequences follow. First, capacity is drawn from a per-subscription, per-region, per-model quota pool, so allocating 200 to a dev deployment takes it away from prod. Second, because deployments are separate resources, you can allocate separately: give the batch job its own deployment with its own capacity and a spike in batch traffic cannot 429 your interactive path.

Provisioned SKUs behave differently again: their capacity unit is provisioned throughput units rather than tokens per minute, they are billed on reserved capacity rather than usage, and they are the reason to read the SKU name before reading the number next to it.

Destroy, soft delete, and the name you cannot reuse

This is the behaviour that turns a routine teardown into a twenty-minute detour, and it is not obvious from the resource documentation. Deleting a Cognitive Services account soft-deletes it: the account is gone from your subscription’s normal view but its name is still reserved. Recreating it with the same name — which is exactly what a destroy followed by an apply does — fails until the soft-deleted account is purged.

The AzureRM provider exposes this in the provider-level features block rather than on the resource, which is why people look for it in the wrong place:

provider "azurerm" {
  features {
    cognitive_account {
      purge_soft_delete_on_destroy    = true
      recover_soft_deleted_cognitive_accounts = true
    }
  }
}

Choose deliberately, because the two settings pull in opposite directions. Purging on destroy makes environments reproducible — tear down, stand up, same names — and removes the safety net that soft delete exists to provide. Recovering soft-deleted accounts is the gentler option: an apply adopts the existing soft-deleted account instead of failing. For a dev environment that is created and destroyed on a schedule, purge. For production, do neither and let a deletion require a human to purge it by hand.

The related question is which edits replace rather than update. The account’s name, kind, location and resource group are all identity, so changing any of them destroys and recreates — and now you are in the soft-delete problem above, with an endpoint URL that changes underneath every client. custom_subdomain_name is in the same category, which is the practical reason to set it at creation rather than discovering you need it later.

Deployments behave better. sku.capacity updates in place, so scaling a deployment’s tokens-per-minute allocation up or down is a cheap, non-disruptive change — which is what makes it reasonable to manage capacity from Terraform at all. The model name and version inside the model block are not; changing them replaces the deployment, and for the duration of that replacement the deployment name your application calls does not exist. On a production path, create the new deployment under a new name, cut traffic over, then remove the old one.

Model versions and the upgrade you did not ask for

Azure OpenAI deployments have an auto-update setting that governs whether Microsoft moves the deployment to a newer model version when the pinned one is retired. Terraform will manage the version you declare, but a service-initiated upgrade changes the deployed version out from under state, and the next plan shows a diff nobody made.

Decide which behaviour you want and encode it, rather than letting the default decide. Pinning a version means you must track retirement dates yourself — Microsoft publishes them per model — and an unattended retirement is a production outage. Allowing auto-update means the model behind your evaluations can change without a deploy, which is its own kind of incident.

Whichever you pick, the deployment-name indirection from the first section is what makes it survivable: a version change is a Terraform edit to one resource, and nothing that calls /openai/deployments/chat/chat/completions needs to know. Keep the account key out of your configuration while you are here — see keeping provider keys out of Terraform state for the write-only-argument pattern, which applies to the AzureRM provider’s ephemeral values in the same way.