Migrating a Multi-Tenant App's Per-Tenant Provider Config

Moving every tenant at once is a single event with a single blast radius, and you only get to be wrong about it once. A per-tenant configuration layer turns that into a sequence you can stop — but only if the resolved configuration reaches the cache key, the credential cache and the metrics, and not just the client constructor.

Resolve once, pass the value down

The layer has three levels: a global default, an optional plan or cohort default, and a per-tenant override. Resolution walks them in order and produces one immutable object — provider, model identifier, region or endpoint, a reference to the credential, the prompt template version, and whatever feature flags change request shape.

Resolve that object exactly once, at the edge of the request, and pass it down. The tempting alternative — a getProviderFor(tenant) helper called wherever a call is made — produces split-brain during a config change: the first call in a request resolves to the old provider, the retrieval call thirty milliseconds later resolves to the new one, and the completion is generated from context embedded by a different model. That failure is intermittent, correlates with nothing in your logs, and takes days to find.

type ResolvedConfig = {
  tenantId: string;
  provider: "provider-a" | "provider-b";
  model: string;
  endpoint: string;
  credentialRef: string;   // secret name + version, never the secret
  promptVersion: string;
  hash: string;            // stable hash of everything above except tenantId
};

Note what is not in it: the API key. The configuration holds a reference — a secret manager name and version — and the secret is fetched by that reference. A configuration table containing key material is a table you cannot log, cannot dump for support, and cannot hand to an auditor.

The resolved config needs an identity

The single highest-value line in this design is the hash field. Compute it from the provider, model, endpoint, credential reference and prompt version — the things that change what the model receives and therefore what it returns — and then put it in four places: every log line, every metric label, the cache key, and the per-call record in your cost attribution table.

With it, “which tenants are on the new configuration” is a GROUP BY, a latency regression is attributable to a configuration rather than to a deploy, and a cache entry cannot outlive the configuration that produced it. Without it, every question in the migration is answered by reading the config table as it is now and assuming it was the same an hour ago.

Keep tenantId out of the hash and carry it separately. The hash answers “what configuration was this”; the tenant answers “whose request was it”. Conflating them means two tenants on identical configurations look like two different configurations, which destroys the grouping that made the field useful.

Credentials and the blast radius of a rotation

Two credential models, with different failure shapes. A shared platform key means one account, one quota and one thing to rotate — and rotating it fails every tenant at once. Per-tenant keys, whether you issue them or the customer brings them, isolate the failure to one tenant but multiply the operational surface.

The blast radius is not set by which model you choose. It is set by where the key is cached. SDK clients are usually constructed once and reused, capturing the credential at construction, so a rotation in the secret manager has no effect until the process restarts. That produces the worst possible ordering: you revoke the old key, the running processes keep presenting it, and every request fails until a deploy rolls through — a full outage created by a routine hygiene task.

Fix it by making the client cache keyed on credentialRef. When the reference changes, the factory builds a new client and evicts the old one; the process picks up the new secret without a restart. Then make the revocation observable: emit the credentialRef in use on every call, and revoke the old version only once you have seen zero requests carrying the old reference for longer than your slowest instance’s config TTL. That gives you a measurable trigger instead of a guess.

Cache keys and rate-limit buckets

Both are derived keys, and both are places where a per-tenant design leaks if the derivation is wrong.

The cache key must include the resolved config hash and the tenant whenever the prompt embeds tenant-specific content. A key computed from the user’s message alone is fine in a single-tenant, single-provider system and becomes a cross-tenant disclosure the moment a tenant-specific system prompt exists, because the part of the input that differs is the part the key ignores. Include the full resolved input, the config hash, and the tenant identifier.

The rate-limit bucket is a property of the credential, not of the tenant. Provider quotas apply to the account the key belongs to, so ten tenants sharing a key share one bucket, and the bucket key in your own limiter must be the credential reference if it is going to model reality. Per-tenant fairness is then a second layer on top — an admission control step that caps each tenant’s in-flight requests before they reach the shared bucket. Skip that layer and you have the noisy-neighbour failure waiting for your first large customer’s backfill.

The migration, in order

1. Introduce the table with everyone on the current provider. No behaviour change, no new provider, nothing to roll back. This step exists to prove the resolution path is correct while the answer is known.
2. Emit the config hash everywhere — logs, metric labels, cost records, cache keys — and let it run for long enough to have a baseline. Dashboards must be able to split by tenant and by hash before anything moves.
3. Add the new provider as a resolvable option behind the same interface, with its own credential reference, and exercise it from a test tenant that has no real traffic.
4. Move your own internal tenant first. You will find the prompt-format differences here rather than in front of a customer.
5. Move one canary tenant with real traffic and watch the per-tenant panels: error rate, latency percentiles, output length, cost per request, and whatever quality signal you have.
6. Move in cohorts, smallest blast radius first, and keep at least one cohort on the old provider until the new one has survived a provider incident. A migration that has never seen the new provider degrade has not been tested.

Rollback is a row with a propagation time

The point of all this is that reverting a tenant is an update to one row. But a row is only a rollback if it takes effect quickly, and configuration is exactly the thing everyone caches. If instances hold the config for five minutes, your rollback time is five minutes plus in-flight requests, and that number is your real incident response time — so publish it rather than discovering it.

Either keep the TTL short enough that the number is acceptable, or add push invalidation and keep the TTL as a fallback. Then test it: change a tenant’s row in staging and measure how long until every instance emits the new hash. That measurement is the one piece of this design that is easy to assume and easy to be wrong about. Moving the router configuration itself has the same property for the same reason.