What Breaks When Two Tenants Use Different Providers

Running two providers behind one codebase is fine until it is not, and when it is not, the symptoms arrive without the word “provider” anywhere in them. These are the five that actually happen, each from the symptom down.

Triage: what the split tells you

Before diagnosing anything specific, label your errors and latency series with both the tenant and the resolved provider. How the anomaly divides tells you which class of bug you have, and it takes minutes rather than the days you will otherwise spend.

• Splits cleanly by provider, across many tenants — a capability or format difference. The code assumes something one side does not do.
• Splits by tenant within a single provider — a configuration problem: a wrong model, a stale prompt version, a credential with different limits.
• Splits by time and hits everyone on one credential — a shared resource: a quota, a bucket, a key.
• Does not split at all — it was not a tenancy bug and the mixed setup is a red herring, which is worth establishing early because mixed setups attract blame.

“A tenant saw another tenant’s answer”

The worst one, and the mechanism is almost always the cache key rather than anything exotic.

A response cache typically starts life keyed on a hash of the user’s message, because in a single-tenant prototype that is the whole input. Then per-tenant system prompts arrive — the customer’s name, their tone rules, an extract of their knowledge base, their retrieved documents — and every one of those goes into the system message and none of them goes into the key. Two tenants who happen to ask the same common question (“what is our refund policy?”, “summarise this”) now collide, and the second one receives a completion generated from the first one’s private context. It is not a race and it is not intermittent: it is deterministic, and it will look to you like a hallucination until someone recognises the other tenant’s data.

A mixed-provider setup makes this both more likely and harder to see. More likely, because the cache is one of the few layers written to be provider-agnostic, so it is the layer where the provider was deliberately left out of the key. Harder to see, because the two tenants are on different providers, so nobody looking at a provider dashboard finds the request that produced the cached answer — it was served from a different vendor entirely.

The key must be a hash of everything that determines the output: the full resolved input including system and tool definitions, the resolved configuration hash from the per-tenant config layer, and the tenant identifier as a namespace prefix. The tenant prefix is belt and braces — with the full input in the key it is redundant, and you want it anyway, because it makes a cross-tenant hit structurally impossible rather than dependent on the completeness of the hash. Add a test that asserts two tenants with identical user messages and different system prompts produce different keys.

“One tenant’s job 429s everybody”

The symptom: for twenty minutes, every tenant on one provider sees elevated latency and a scatter of failures. Nothing was deployed. One large customer happened to start a backfill.

The mechanism has two halves. First, provider rate limits attach to the account the credential belongs to, not to your notion of a tenant. Ten tenants sharing a key share one bucket, so one tenant’s burst consumes the shared allowance and everyone else’s requests are rejected — the tenant causing the problem is not the tenant experiencing it.

Second, and this is the half that turns twenty minutes into an hour, your retry layer amplifies it. Every rejected request from every well-behaved tenant is retried with backoff, so the moment the noisy job finishes there is a backlog of retries that immediately resaturates the bucket. The system stays degraded after the cause has stopped, which is why the timeline never quite lines up with the backfill and why people go looking for a second cause.

Three fixes, in order of how much they help. Add per-tenant admission control in front of the shared bucket — a concurrency cap or weighted queue per tenant — so the rejection lands on the tenant generating the load and everyone else never sees a 429. Honour the retry-after value the provider returns rather than your own backoff curve, and add full jitter so retries do not re-cluster. And move the heavy tenant to its own credential, which converts a shared-fate problem into a per-tenant quota problem you can bill for.

One shared-fate case worth naming separately because it looks identical from the outside: a credential rotation. Every tenant on that credential fails simultaneously and tenants on the other provider are untouched, which reads exactly like a provider outage. The distinguishing signal is the status code — authentication errors rather than 429s or 5xx — and the cure is in the credential caching design rather than in the limiter.

“Parse errors, but only for some tenants”

Shared prompt templates are where one provider’s quirks get encoded as though they were universal. The template works, so nobody examines it; it works because it was tuned against one model’s behaviour.

The usual culprits: a template that relies on prefilling the start of the assistant’s reply to force a JSON opening brace, which some models do not accept at all; a stop sequence chosen because one model’s output reliably contained a particular marker; an instruction that only produces valid JSON because the request also sets a JSON-output mode that the other provider expresses differently or not at all; or a system message that is a first message in one API and a dedicated top-level parameter in another, so it silently carries a different weight.

Each shows up as malformed output confined to a subset of tenant identifiers, which is why it gets misfiled as a data problem with “those customers’ documents”. Group your parse failures by provider once and the shape is obvious. The fix is to version prompt templates per provider rather than sharing one, accept the duplication, and pin the version in the resolved config so a template change is attributable — prompt portability covers what does and does not carry across.

“The cost dashboard stopped adding up”

Usage reporting is where two providers are least alike, and a per-tenant cost table that sums a column called tokens across both is adding numbers that do not mean the same thing.

• The field names differ and so do the boundaries. Prompt and completion versus input and output is the easy part; whether cached input is reported inside the input count or as a separate figure, and whether reasoning tokens are broken out or folded into output, changes what any total means.
• Streaming may not report usage at all by default. At least one major API requires an explicit opt-in on the request before usage appears in the final chunk of a streamed response — check the streaming section of the reference for each provider you serve. If you miss it, your streaming tenants show near-zero cost and your totals are quietly wrong in the direction nobody investigates.
• Prices change on different days, so a table storing only money has no way to be recomputed and a table storing only tokens has no way to be totalled.

Store both, at write time: the raw usage object exactly as the provider returned it, in a provider-labelled column, plus a money figure computed when the record is written together with the identifier of the price list used. Then a total is a sum of money, a per-provider comparison is a query over raw usage, and a price change is a new price list version rather than a retroactive rewrite of history. Cost attribution covers the general shape of that table; the only thing mixed tenancy adds is that the provider column is no longer constant, which is exactly the assumption every dashboard built before the second provider quietly made.