Auditing Every Hardcoded Provider Assumption in a Codebase

The hard part of a provider migration is not the code that calls the provider. It is the code three layers away that assumes what the response looked like, and an audit exists to find that before somebody commits to a date.

Why an inventory before an estimate

Teams estimate migrations by counting client call sites, and that number is almost always small and almost always wrong. A single call site produces assumptions that spread: a helper that reads choices[0], a metrics job that sums a usage field by name, a test fixture recorded in one wire format, an alert threshold tuned to one provider’s error taxonomy. None of those import the client.

The audit below is nine greps. It takes an afternoon and it produces a list you can sort, assign and argue about. Run it before the design discussion, not after, because the design that is right for forty scattered assumptions is different from the one that is right for four.

The nine passes

Run each of these across the whole repository including tests, fixtures, notebooks, infrastructure code and documentation. The examples use ripgrep; adjust the file-type filters to your stack.

# 1 — client construction and configuration
rg -n "OpenAI\(|AsyncOpenAI|openai\.api_key|ANTHROPIC_API_KEY|base_url"

# 2 — response shape access
rg -n "choices\[0\]|\.choices|message\.content|\.tool_calls|finish_reason"

# 3 — usage and cost fields
rg -n "prompt_tokens|completion_tokens|total_tokens|input_tokens|output_tokens"

# 4 — parameter names that are renamed or unsupported elsewhere
rg -n "max_tokens|max_completion_tokens|logit_bias|logprobs|top_logprobs|seed|response_format|presence_penalty"

# 5 — provider-specific exception types and status handling
rg -n "openai\.(RateLimitError|APIError|BadRequestError)|except openai|status_code == 429"

# 6 — tokenizer coupling
rg -n "tiktoken|cl100k|o200k|encoding_for_model|count_tokens"

# 7 — model identifiers, wherever they hide
rg -n --glob '!node_modules' "gpt-|claude-|gemini-|mistral-|-turbo|-latest"

# 8 — streaming assumptions
rg -n "delta\.|data: \[DONE\]|text/event-stream|iter_lines"

# 9 — recorded fixtures and cassettes
rg -l "system_fingerprint|chatcmpl-|\"object\": \"chat.completion\"" tests/

Pass 7 is the one that surprises people. Model identifiers turn up in alerting rules, in seed data, in a comment that documents a threshold, in a dashboard query, and in the prompt itself where somebody wrote “you are GPT-4”. That last one is not a code change; it is a content change, and it is invisible to every other pass.

Pass 9 matters because recorded fixtures encode the old wire format permanently. A migration that updates the code and not the cassettes produces a green test suite that proves nothing, and it will stay green through the outage.

Triaging by failure mode

Sort every hit into one of four buckets. The order is deliberate: it is increasing danger, not increasing effort.

• Fails to compile or import. A removed class, a renamed module. Free — the toolchain finds these for you and you can stop tracking them.
• Raises at runtime on the first call. An unsupported parameter rejected with a 400, an attribute that does not exist on the new response object. Cheap, because one smoke test in staging surfaces the whole set.
• Raises at runtime on some inputs. The empty-candidate case when a prompt is filtered, a second tool call your loop drops, a token limit only long inputs reach. These need deliberate test cases because ordinary traffic will not produce them on the day you look.
• Never raises and is wrong. The expensive bucket. Everything below.

The silent ones, in detail

Usage field names that do not exist on the new response. A metrics collector reading a token count by name gets nothing, records zero, and your cost dashboard shows a beautiful reduction. Nobody investigates a cost graph that goes down. Assert non-zero usage in the adapter and alarm on a sustained zero.

Parameters the endpoint ignores. An OpenAI-compatible endpoint that accepts unknown fields will take your bias map, your seed and your penalty settings and discard them. There is no error and no field in the response saying so. The detection is deliberate: send a request with a parameter set to an absurd value and confirm the output changes.

Token counts from the wrong tokenizer. Code that budgets context or estimates cost with one vendor’s tokenizer, against another vendor’s model, is wrong by a factor that varies with the content — most sharply on non-Latin scripts and on code. It fails as truncation you attribute to the model. The library covers the mechanism in the tokenizer mismatch bug and the billing consequence in token count mismatch.

Finish-reason branches that no longer match. A comparison against a string literal that the new provider never emits takes the else branch forever. Grep for the literals, not the field, and make the default branch log loudly rather than fall through.

Retry policies keyed to one error taxonomy. A retry rule that matches an exception class from one SDK matches nothing from another, so requests that used to be retried now fail on the first attempt. The reverse also happens: a broad catch retries a deterministic 400 forever and multiplies your bill.

Prompts naming the model. Instructions that reference the model by name, or few-shot examples written in one model’s characteristic register, degrade quietly. No test catches this and no grep for code does either — pass 7 over prompt files is what finds it.

Turning the inventory into work

1. Put every hit in one table with four columns: file, pass number, failure bucket, owner. Do not summarise yet — the row count is the estimate.
2. Delete the rows that are dead code. On most audits this is a meaningful fraction, and it is the cheapest work available.
3. Decide, per bucket-four row, whether the fix is a rename, a shim, or a behaviour change your product owner has to accept. The third category is the one that needs a decision from outside the engineering team, and it is the one that delays cutovers.
4. Write the adapter boundary before changing any caller, so that the fixes have somewhere to land. The related page on writing a provider adapter layer covers the shape.
5. Re-run all nine passes after the migration. The number that should be zero outside the adapter is passes 2, 3, 5 and 8; anything still matching elsewhere is an assumption that escaped, and it is far cheaper to find it now than during the next migration.

The audit is worth keeping as a script rather than an afternoon. Run it in continuous integration with a threshold, and the count of provider assumptions outside the adapter becomes a number that can only go down. The related discipline is in provider-agnostic AI code.