Building an Internal Model Gateway

A model gateway is a service every model call in an organisation goes through, so that authentication, routing, quota, failover, logging and key custody happen once instead of in forty codebases. It is a straightforward piece of engineering with two genuinely hard parts — correct streaming and distributed quota — and a maintenance burden that is larger than the build. This page describes the architecture, and then states the costs.

What a gateway is for

The case for one becomes obvious at a particular size. Once several teams call models independently, five things are true at once: nobody can say what the organisation spends, provider keys are distributed across services and CI systems, switching providers means editing every repository, there is no consistent record of what was sent to a third party, and every team has written its own retry logic with different bugs.

A gateway addresses all five by being the single egress point. That is its real definition: not a proxy, but the place where organisational policy about model use is expressed once and enforced.

It also introduces a hard dependency — a component that, when down, stops every AI feature in the company. That is the central trade, and it means a gateway must be engineered to a higher availability standard than most of the services behind it. Design for it to fail open where policy permits: a client library that can call a provider directly with a break-glass credential when the gateway is unreachable turns a total outage into a degraded mode, at the cost of a credential that must be audited.

The seven components

The request path through them, in order: authenticate, resolve the alias, check quota, select a route, attach the provider credential, call, stream the response back while counting tokens, record the audit entry, decrement quota with actual usage. Everything on that path is latency the caller pays, so the target is a few milliseconds of gateway overhead excluding the provider call — which mostly means the quota check must be fast and the audit write must be asynchronous.

Routing and the model alias

The most valuable single feature is indirection: callers ask for a capability, not a vendor’s product name.

# routes.yaml — the routing table is configuration, reviewed like code.
aliases:
  chat-default:
    routes:
      - provider: vendor-a
        model: model-a-mid
        weight: 100
        max_context: 128000
      - provider: vendor-b            # failover, tried on route failure
        model: model-b-mid
        weight: 0
    constraints:
      residency: eu                   # only routes whose region satisfies this
      max_cost_per_mtok_output: 2.00

  extract-cheap:
    routes:
      - provider: vendor-c
        model: model-c-small
        weight: 100
    require:
      structured_output: true         # a capability, not a product feature name

  code-review:
    routes:
      - provider: self-hosted
        model: local-model-a
        endpoint: http://inference.internal:8000/v1
        weight: 100
      - provider: vendor-a
        model: model-a-large
        weight: 0                     # overflow when the local pool is saturated

Three properties follow. A provider migration is a pull request rather than a coordinated change across every team. A canary is a weight change, which makes canary deploys for model changes possible without touching any caller. And residency becomes enforceable centrally rather than by convention — a route that does not satisfy the constraint is not selectable, which is a much stronger guarantee than a policy document.

Keep the routing logic boring. Health-aware selection — skip a provider whose circuit breaker is open, prefer the cheapest route that satisfies the constraints — is worth having. Adaptive routing that learns from latency in real time is a research project that will produce incidents you cannot reproduce; if you want it, ship it behind a flag with the static table as the fallback.

Quota, and why it is the hard one

Quota is the component that looks trivial and is not, for a reason specific to model calls: you do not know the cost of a request until after it has completed. Output token count is unknown at admission. So the naive design — check a counter, then decrement it — cannot work, and the honest design is a reservation.

1. Estimate at admission. Input tokens are countable before the call; output is bounded by max_tokens. Reserve the worst case: input plus max_tokens, priced at the route’s rates.
2. Reject if the reservation exceeds the remaining budget. This is what makes a spend cap a real cap rather than a report — the request never happens.
3. Reconcile on completion. Release the difference between the reservation and actual usage. Most responses are far shorter than max_tokens, so the reservation is conservative and the reconciliation matters.
4. Expire reservations. A crashed gateway replica leaves reservations held forever, and a caller’s budget silently shrinks until somebody investigates. A TTL on every reservation is not optional.

-- Sketch of an atomic reserve, in a shared store with server-side scripting.
-- KEYS[1] = budget key for this caller and window
-- ARGV[1] = reservation id, ARGV[2] = estimated micros, ARGV[3] = ttl seconds
local remaining = tonumber(redis.call('HGET', KEYS[1], 'remaining') or '0')
local est = tonumber(ARGV[2])
if remaining < est then
  return {0, remaining}                      -- refuse; caller sees 429
end
redis.call('HINCRBY', KEYS[1], 'remaining', -est)
redis.call('HSET', KEYS[1] .. ':res', ARGV[1], est)
redis.call('EXPIRE', KEYS[1] .. ':res', tonumber(ARGV[3]))
return {1, remaining - est}

-- On completion: release (est - actual) back to 'remaining' and delete the
-- reservation. A sweeper releases reservations whose TTL passed, so a
-- crashed replica does not permanently consume budget.

Two further decisions. Failure mode when the quota store is unreachable — fail open and risk unbounded spend, or fail closed and take an outage. State it explicitly per caller class; the usual answer is fail closed for spend caps and fail open for rate limits, because the consequences differ. And where the limit is enforced: per replica is easy and wrong by a factor of your replica count; centrally is correct and adds a round trip to every request. A common compromise is a local token bucket for rate limits, refreshed from the central store, with the spend cap always checked centrally.

Streaming is where implementations break

A gateway that buffers the whole response and forwards it has destroyed the product experience: time to first token becomes total generation time. Streaming through is mandatory, and it brings four requirements that are easy to get wrong.

• No buffering anywhere in the path. Disable response buffering in every reverse proxy in front of the gateway. This is the single most common cause of “streaming does not work through our gateway” and it is a configuration line, not a code change.
• Backpressure must propagate. If the client reads slowly, the gateway must stop reading from the provider rather than buffering in memory. A gateway that accumulates unread chunks for a thousand concurrent slow clients runs out of memory in a way that looks like a leak.
• Cancellation must propagate. When the client disconnects, the provider call must be aborted. Otherwise you continue paying for generation nobody will read, which is pure waste and is invisible in every dashboard except the bill.
• Usage arrives at the end, or not at all. Token counts typically come in a final event, and some providers omit them under some conditions. You need a fallback: count tokens locally with the right tokeniser, and mark such records as estimated so your cost report does not silently mix measured and estimated figures.

Add error mapping to that list. Providers signal failure differently — an HTTP status before the stream, an error object inside the stream, or a truncated stream with no signal at all. All three must become one canonical error taxonomy for callers, or every client team re-implements the mapping and normalising API errors becomes forty slightly different attempts.

Key custody and audit

Custody is the most under-appreciated benefit. With a gateway, provider keys exist in exactly one system; callers hold a gateway credential that is scoped, revocable and attributable. Revoking one team’s access is a single operation with no provider involvement, and a provider key rotation is one update rather than a fleet-wide roll — the procedure in rotating a provider key without downtime becomes an internal operation.

Audit is where policy decisions have to be made explicitly, because the useful record and the risky record are the same record.

# Two tiers, because they have different retention and access rules.

# Tier 1 — metadata. Always recorded. Low risk, long retention.
{"ts": "...", "request_id": "...", "caller": "svc-search", "user_hash": "...",
 "alias": "chat-default", "provider": "vendor-a", "model": "model-a-mid",
 "input_tokens": 4180, "output_tokens": 260, "cached_input_tokens": 3900,
 "cost_micros": 198, "ttft_ms": 410, "total_ms": 2410,
 "status": "ok", "route_attempt": 1, "quota_key": "team:search:2026-08"}

# Tier 2 — content. Prompt and completion. Recorded only where policy allows.
# Separate store, separate access control, shorter retention, encrypted,
# redacted for personal data before write, and excluded entirely for callers
# whose data class forbids it.

Splitting the tiers is what makes the metadata usable by everyone while the content stays restricted. Get the retention decision written down before launch: content logs are the highest-value debugging artefact you will have and the highest-risk data you will hold, and the two facts do not resolve themselves. Audit logs for AI systems and PII in LLM logs cover the specifics, and cost allocation is downstream of tier one.

Build or buy, honestly

The initial build is not the decision. A competent team can have routing, key custody and logging working in a couple of weeks. The decision is about what comes after, so here is the ongoing work enumerated rather than summarised.

What building actually costs

• Provider API drift. Every provider changes request shapes, adds parameters, deprecates models and alters error semantics on its own schedule. Each adapter is a small permanent maintenance stream, and the number of streams is the number of providers you support.
• The price table. Cost telemetry is only correct if prices are current, per model, per token class — input, cached input, output, reasoning tokens, images, audio. Somebody must own updating it, and a stale table produces confidently wrong cost reports, which is worse than none.
• Token accounting for non-text. Image and audio token counting differs per provider and per model. Getting it wrong is invisible until a bill arrives.
• Availability of the gateway itself. It is now a single point of failure for every AI feature, which means an on-call rotation, an SLO, capacity planning and its own runbooks. This is the cost people most consistently omit.
• Streaming correctness, per provider. The four requirements above have to hold for each adapter, and the failure modes — memory growth under slow clients, orphaned generations after disconnect — appear only under production concurrency.
• Quota state under concurrency. Reservations, reconciliation, expiry, and the failure mode when the store is unreachable. This is genuinely distributed-systems work and it is where the subtle bugs live.
• Audit retention and data requests. Deletion requests, retention schedules, access controls, and the ability to answer “what did this user send in March?” correctly.
• New capabilities. Tool calling, structured outputs, multimodal inputs, caching semantics, reasoning-token accounting. Each arrives from providers on their timetable and each needs adapter work before your callers can use it — and callers will ask immediately.

A reasonable planning figure, stated as an assumption rather than a measurement: the initial build is weeks and the steady state is a fraction of an engineer indefinitely, rising with the number of providers and capabilities supported. Substitute your own numbers; the shape of the estimate — small build, permanent tail — is the part that generalises.

When building is right

• Routing rules that are specific to your organisation — data classification driving model selection, per-department policy, a customer-specific model. A general product will not express these and forcing it to is worse than building.
• An existing internal platform to attach to. If identity, policy and audit already exist and the gateway is a thin layer over them, the build is much smaller than the list above suggests.
• Strict residency or air-gap requirements that no external service satisfies. See air-gapped and offline deployments.
• Mostly self-hosted models. If you are routing between your own inference pools rather than between vendors, most of the adapter maintenance disappears and what remains is a load balancer with quota — a much smaller thing.
• Model routing is your product. Then it is not infrastructure, it is the thing you are building, and the question does not arise.

When buying is right

• The requirement is standard. One API, several providers, quota, audit, failover, cost telemetry. This is a commodity and building it means maintaining a commodity.
• The team is small. A permanent fraction of an engineer is a large fraction of a small team, and it is the fraction not building your product.
• You need provider breadth. Supporting fifteen providers is fifteen maintenance streams, and breadth is the dimension where a dedicated product has the clearest advantage.
• Speed matters more than control for now. Note that this is reversible in one direction: an alias-based interface that you own means the gateway behind it can be replaced later, whichever way you go. Design your callers against your own alias namespace rather than a vendor’s model identifiers, and this decision stops being permanent.

The middle path is real and often the right one: buy for breadth of external providers, run your own thin layer for the organisation-specific policy, and keep the alias namespace yours.