Switching LLM Provider: The Incompatibility Checklist

A provider switch fails in three waves. The request surface breaks first and loudly, error handling breaks second and quietly, and behaviour breaks third — weeks later, in production, on requests that return HTTP 200. Planning for the first wave and not the other two is the usual mistake.

The order things break in

Both destinations may advertise an OpenAI-compatible endpoint. That claim covers the request and response envelope and almost nothing else, so the compatibility work is not where the SDK is.

Budget accordingly: wave 1 is hours, wave 2 is days, and wave 3 is the one that decides whether the migration was worth it.

Layer 1: the request surface

Work through this list against both providers’ current documentation before writing any code. Every item has bitten someone.

• Token-limit parameter naming. Some APIs take max_tokens, some take max_completion_tokens, and some accept both. If your code forwards whichever it finds first, a request carrying both can reserve against one number and be billed against the other. Take the larger of the two when reasoning about cost.
• Silently dropped parameters. The dangerous case is not rejection, it is acceptance. A provider that ignores logprobs or parallel_tool_calls rather than refusing them returns a fully-billed 200 and your downstream code fails on a missing field. Refuse unsupported parameters at your own boundary instead of forwarding and hoping — silently dropped parameters covers the class.
• Extra fields. Strict OpenAI-compatible servers return 400 on an unknown field. If your client adds vendor-specific keys for one provider, they must be stripped for the others, and the stripping list must be one shared list rather than one per endpoint.
• System-message handling. Providers differ on whether a system message may appear anywhere other than first, whether multiple are allowed, and whether it is merged into the first user turn.
• Tool-calling schema dialect. Nesting depth, whether additionalProperties: false is required, whether unions are supported, and whether the arguments arrive as a JSON string or an object.
• Tokenisation. Different tokenisers mean the same text is a different number of tokens, so a prompt that fits one provider’s window can exceed another’s at the same character count. See comparing tokenisers.

Layer 2: errors and limits

The classification problem is the expensive part of this layer. Providers do not agree on which status code carries which meaning, and two disagreements matter more than the rest.

• An exhausted account balance. One provider may report it as 400, another as 429. Under a status-only classifier the first is treated as a caller error and returned to the user; the second is retried with backoff against an account that will refuse every attempt. Neither is a statement about the request, and both need the response body read. The full classifier is in production LLM error classes.
• Rate-limit signalling. Header names, whether a retry-after is supplied, whether limits are per-minute tokens or per-minute requests or both, and whether they are per-key or per-organisation.
• Streaming terminators. Whether a usage object arrives in the final chunk, whether a sentinel line closes the stream, and what a mid-stream error looks like. If your billing reads usage only from a terminal chunk, a cancelled stream on a provider that does not send one bills you nothing while the provider bills you in full.
• Timeout semantics. Whether the provider closes an idle connection or holds it, and whether the first token or the whole response is what your deadline should cover.

Layer 3: behaviour, which no test catches

Wave 3 is where migrations actually fail, because nothing in the request or the response is wrong. The same prompt simply produces different text.

• Format drift. One model wraps JSON in a code fence, another does not. One prefixes with a sentence of preamble. Both are valid completions and only one parses.
• Instruction-following differences. Negative instructions, length limits and “answer only with” constraints are honoured to different degrees. A prompt tuned against one model is, in effect, over-fitted to it — prompt portability is the general case.
• Refusal thresholds. The same input can be answered by one provider and refused by another, which shows up as a spike in empty completions rather than as an error.
• Tool-calling eagerness. Some models call a tool when they should answer directly, and some do the reverse. In an agent loop this changes the step count, and therefore the cost, more than the token price does.

A differential harness

Wave 3 cannot be reasoned about, only sampled. The following runs your own real prompts against both providers and reports the differences that matter, before any traffic moves. It uses only the standard library plus your existing client; substitute your own call function.

# diff_providers.py — run your own prompts against two providers
# Python 3.11. Replace call() with your own client for each provider.
import json, statistics, sys, time

PROMPTS = json.load(open("prompts.json"))   # [{"id":..., "messages":[...]}, ...]
N = 5                                        # samples per prompt per provider
FENCE = chr(96) * 3                          # the markdown code-fence marker

def call(provider, messages):
    """Return (text, usage_dict, seconds). Implement with your own client."""
    raise NotImplementedError

def sample(provider, messages):
    out = []
    for _ in range(N):
        t0 = time.time()
        text, usage, _ = call(provider, messages)
        out.append({
            "text": text,
            "chars": len(text),
            "out_tokens": usage.get("completion_tokens"),
            "in_tokens": usage.get("prompt_tokens"),
            "seconds": time.time() - t0,
            "parses": parses(text),
            "fenced": text.lstrip().startswith(FENCE),
            "empty": text.strip() == "",
        })
    return out

def parses(text):
    t = text.strip()
    if t.startswith(FENCE):
        t = t.split("\n", 1)[-1].rsplit(FENCE, 1)[0]
    try:
        json.loads(t)
        return True
    except Exception:
        return False

def summarise(rows):
    return {
        "parse_rate": sum(r["parses"] for r in rows) / len(rows),
        "fence_rate": sum(r["fenced"] for r in rows) / len(rows),
        "empty_rate": sum(r["empty"] for r in rows) / len(rows),
        "median_out_tokens": statistics.median(
            r["out_tokens"] for r in rows if r["out_tokens"] is not None),
        "median_seconds": statistics.median(r["seconds"] for r in rows),
    }

a, b = sys.argv[1], sys.argv[2]
for p in PROMPTS:
    ra, rb = sample(a, p["messages"]), sample(b, p["messages"])
    sa, sb = summarise(ra), summarise(rb)
    flags = []
    if abs(sa["parse_rate"] - sb["parse_rate"]) > 0.05: flags.append("PARSE")
    if abs(sa["fence_rate"] - sb["fence_rate"]) > 0.20: flags.append("FENCING")
    if abs(sa["empty_rate"] - sb["empty_rate"]) > 0.02: flags.append("REFUSAL")
    if sb["median_out_tokens"] > 1.3 * sa["median_out_tokens"]: flags.append("VERBOSITY")
    print(p["id"], json.dumps({a: sa, b: sb, "flags": flags}))

Four flags, and each maps to a real cost. PARSE means downstream code breaks. FENCING means it breaks in one specific, easily repaired way. REFUSAL means users lose answers they used to get. VERBOSITY means the per-token price comparison that justified the migration was computed on the wrong token count — a provider 30% cheaper per token that emits 40% more tokens is more expensive.

Run it on at least fifty of your own real prompts, not on synthetic ones. The prompts that break are almost always the long, messy, production-shaped ones.

The cutover

1. Normalise before you migrate. Put one adapter boundary in front of both providers and move the current provider behind it first, with no behaviour change. A migration and a refactor at once is two unknowns.
2. Shadow before you switch. Send a copy of real traffic to the new provider, discard the responses, and compare the aggregates from the harness. It costs the new provider’s token price and nothing else.
3. Ramp by percentage, not by feature. Percentage ramping keeps the comparison population comparable; feature-by-feature ramping compares different traffic and tells you nothing.
4. Keep the old route warm for a fortnight. The wave-3 problems arrive after the celebration. A rollback that requires re-adding a deleted adapter is not a rollback.
5. Re-run your evaluation set on the new provider before, not after. If you do not have one, build a small one — fifty labelled examples is enough to catch a wave-3 regression and is a day of work.