Hybrid Architecture: Local for Most, API for Hard Requests

Most request streams are not uniformly hard. If yours is not, you can serve the easy majority on a model you control and escalate the difficult remainder — and the arithmetic on that is unusually favourable, right up until the complexity costs more than it saves.

The shape that makes this work

The whole idea rests on one empirical property of your traffic: difficulty is distributed, not constant. In most applications a large majority of requests are routine — a classification, a short summary, a template fill — and a minority genuinely need frontier capability.

Check this before designing anything. Sample two hundred real requests and label each as one an open mid-size model could handle or one it could not, using the task-shape test from the small-models page: information present in the context, short dependency chain, constrained output. If ninety per cent land in the first bucket, a hybrid is worth building. If half do, it is not — the escalation rate is too high to pay for the extra system, and you should simply pick one model.

Routing rules that survive contact

Ordered from most robust to least. The first three are the ones that keep working; the last two are where people start and where the trouble is.

• Route by task type. Different endpoints, different models, decided at design time. Classification and extraction go local; open-ended generation and multi-step tool use go to the API. No runtime decision, no misrouting, trivially debuggable. Start here.
• Route by an input feature you can compute. Prompt length, presence of code, number of constraints in the request, whether retrieval returned anything. Cheap, deterministic, explainable.
• Try local, escalate on a mechanical failure. The strongest general pattern. Run the local model, then check the output with something that cannot lie: does the JSON parse, does the code compile, are the required fields present, did the model emit a refusal or a hedge. Escalate only on failure. Correctness of the check, not a prediction of difficulty, drives the routing.
• Escalate on self-reported confidence. Asking the model whether it is sure. Weak — models are poorly calibrated, and small ones are worse. Usable as one signal among several, never alone.
• A learned router. A classifier predicting which model will succeed. Can work well, requires labelled data, and adds a component that itself needs monitoring. Do this last, if the simpler rules leave real money on the table.

The escalate-on-failure pattern deserves emphasis because it is the only one that degrades gracefully. When the local model gets better, fewer requests escalate and nothing needs changing. When it gets worse — a bad quantisation, a template regression — escalation rises and your monitoring shows it immediately.

The cost model

e            escalation rate (fraction of requests going to the API)
c_local      cost per request locally
c_api        cost per request at the frontier

# try-local-then-escalate: escalated requests are paid for twice
hybrid = c_local + e * c_api

# versus everything at the API
saving = c_api - hybrid = (1 - e) * c_api - c_local

# worth building only while
#     e < 1 - (c_local / c_api)

Read that last line carefully, because it is the whole decision. If local inference costs a tenth of the API per request, the hybrid pays while fewer than ninety per cent of requests escalate — which is almost always. If local costs half as much, escalation must stay below fifty per cent. And a hybrid built on the try-first pattern pays for the local attempt on every escalated request, so a high escalation rate is worse than not having built it.

Two terms the arithmetic omits and you should not. Latency: an escalated request pays both models sequentially, so your p95 is the sum, not the maximum. And engineering: two model paths means two sets of prompts, two evaluation runs and two failure modes, which is a standing cost that no per-request saving shows.

Implementing it

from openai import OpenAI

local = OpenAI(base_url="http://localhost:8000/v1", api_key="local")
remote = OpenAI(api_key=os.environ["API_KEY"])

def answer(messages, schema):
    try:
        out = local.chat.completions.create(
            model="local-8b", messages=messages, temperature=0,
        ).choices[0].message.content
        result = validate(out, schema)      # raises on any defect
        metrics.inc("served.local")
        return result
    except (ValidationError, APIError, TimeoutError) as exc:
        metrics.inc("escalated", reason=type(exc).__name__)
        return validate(remote.chat.completions.create(
            model="frontier-model", messages=messages, temperature=0,
        ).choices[0].message.content, schema)

• Both sides speak the same API, which is what makes this a base-URL change rather than an integration. Local batching servers expose an OpenAI-compatible interface precisely so this works.
• Escalate on infrastructure failure too. The local box being down is an escalation, not an outage — the same code path gives you failover for free, which is often the better half of the benefit.
• Count escalations by reason, not just in total. A rise in schema failures and a rise in timeouts mean completely different things.
• Set a tight timeout on the local attempt. A slow local model must not make an escalated request slower than going straight to the API would have been.
• Adapt the prompt per side. The same prompt rarely performs equally on both; the porting page covers what to change.

What goes wrong

• Escalation rate drifts and nobody notices. Input distribution changes, the local model starts failing more, costs quietly converge on the all-API case. Alert on the rate, not just on spend.
• Inconsistent outputs between paths. Two models mean two styles and two formats. If users can tell which one answered, constrain both to the same schema and post-process identically.
• Only the easy cases are ever tested locally. Because the hard ones escalate, your local model’s evaluation set slowly becomes the easy slice and you lose the ability to tell whether it could handle more.
• Privacy leaks through the escalation path. If the reason for local inference was that data must not leave, then escalating sends exactly the hardest — and often most sensitive — requests to a third party. In that case escalation must be a queue for a human, not a call to an API. This is the single most important thing to get right, and it is easy to get wrong by accident.
• The complexity outlives its justification. Open models improve; a year later the escalation rate may be low enough that the second path is dead weight, or the first path is good enough alone. Re-run the sampling exercise annually.