Auditing a Codebase for Every Place a Model Name Is Hardcoded

“Change the model name” sounds like one edit. The audit exists to find out how many edits it actually is, and — more usefully — to find the ones that will not fail loudly when you miss them.

Four things to inventory, not one

Searching for model strings alone produces a comfortable, wrong answer. A model reference is only one of four kinds of provider coupling, and the other three are the ones that survive a rename and break later:

• Model identifiers. The literal strings passed as model. These fail loudly on cutover — a wrong one usually comes back as a 404-shaped model not found error — which makes them the least dangerous category.
• Base URLs and endpoint paths. A hostname baked into a client constructor, or a path assembled by hand. These fail loudly too, unless the constant is only read on a code path you do not exercise in staging.
• Token budgets. Numbers that encode a context window or an output cap. A budget sized for one model’s window and reused against a smaller one truncates silently, and truncation produces plausible output rather than an error. This is the category that costs you a week of confused debugging after the cutover.
• Provider-specific error handling. Exception classes imported from one vendor’s SDK, string matches on error codes, and branches on the completion-stop field — which is called finish_reason in OpenAI’s Chat Completions responses and stop_reason in Anthropic’s Messages responses, with different value vocabularies behind each. A retry branch that only catches one vendor’s rate-limit exception becomes a retry branch that catches nothing.

Search patterns you can run today

These use ripgrep. Run them from the repository root. Each is written to over-match deliberately: a false positive costs you two seconds of reading, and a false negative costs you an incident.

Model identifiers. Search by vendor prefix rather than by exact string, because the exact strings are what you do not know:

rg -n --hidden --glob '!node_modules' --glob '!.git' \
  -e 'gpt-[0-9a-z]' \
  -e 'claude-[0-9a-z]' \
  -e 'gemini-[0-9]' \
  -e 'llama-?[0-9]' \
  -e 'mistral-|mixtral-|deepseek-|qwen[0-9]?-' \
  -e 'text-embedding-' \
  -e 'whisper-|tts-1|dall-e'

Then catch the ones whose names you could not guess, by searching for the shape of the assignment instead of the value:

rg -n -e 'model\s*[:=]\s*["\x27]' -e '"model"\s*:\s*"' \
  --glob '!node_modules' --glob '!*.lock'

Base URLs. Hostnames, and the parameter names that carry them:

rg -n -e 'api\.openai\.com' \
  -e 'api\.anthropic\.com' \
  -e 'generativelanguage\.googleapis\.com' \
  -e 'openai\.azure\.com' \
  -e 'bedrock-runtime\.' \
  -e 'base_?url|baseURL|BASE_URL|api_base|endpoint\s*[:=]'

Token budgets. Both the parameter names and the bare integers that encode a window. The parameter has been spelled three ways across two of OpenAI’s own APIs alone — max_tokens in Chat Completions, max_completion_tokens as its replacement there, and max_output_tokens in the Responses API — while Anthropic’s Messages API requires max_tokens on every call:

rg -n -e 'max_tokens|maxTokens|max_completion_tokens|max_output_tokens' \
  -e 'context_?window|contextWindow|MAX_LEN|TOKEN_LIMIT|token_budget' \
  --glob '!node_modules'

# bare integers that are almost always a window or a cap
rg -n -e '\b(2048|4000|4096|8000|8192|16384|32000|32768|100000|128000|200000|1000000)\b' \
  --glob '!node_modules' --glob '!*.lock' --glob '!*.min.*'

The integer search is the noisy one and it is the one worth reading in full. A 4096 in a buffer size is irrelevant; a 4096 being subtracted from a prompt length is a truncation policy tied to a model you are about to stop using. The distinction between the window and the output cap is a common source of these — see context window versus max tokens.

Provider-specific error handling. SDK exception classes, wire-level error codes, and stop-reason branches:

rg -n -e 'openai\.(RateLimitError|APIStatusError|APIConnectionError|APITimeoutError|BadRequestError|AuthenticationError)' \
  -e 'anthropic\.(RateLimitError|APIStatusError|BadRequestError|OverloadedError)' \
  -e 'context_length_exceeded|insufficient_quota|invalid_request_error|rate_limit_error' \
  -e 'finish_reason|stop_reason|refusal' \
  -e 'status_code\s*==\s*429|response\.status\s*===\s*429'

Add one more that finds nothing in a well-behaved codebase and everything in a real one: the branch that reacts to a truncated completion by comparing a string.

rg -n -C2 -e '["\x27](length|max_tokens|stop|end_turn|tool_calls|tool_use|content_filter|stop_sequence)["\x27]'

Those literals are the two vocabularies. OpenAI’s finish_reason takes values including stop, length, tool_calls and content_filter; Anthropic’s stop_reason takes end_turn, max_tokens, stop_sequence and tool_use. Anything comparing against one set will read as “not truncated” when handed the other, which is precisely the silent failure. The per-value treatment of finish_reason covers what each one obliges you to do.

Where the grep misses

Two categories of in-repository hit escape the searches above, and both are ordinary rather than exotic.

Constructed strings. A name assembled from a family and a version, or interpolated from a date suffix, matches no literal pattern. Search for the concatenation instead: a template or format string whose surrounding text mentions a model family, and any lookup table keyed by a short internal alias. The alias table is the useful find — it usually means somebody already started centralising and stopped halfway.

Non-source files. Fixtures, cassettes, snapshots and golden files carry model names inside recorded HTTP bodies, and a default ripgrep run may skip some of them or the repository’s .gitignore may exclude them entirely. Re-run with --no-ignore --hidden restricted to those directories:

rg -n --no-ignore --hidden \
  -g '**/fixtures/**' -g '**/cassettes/**' -g '**/__snapshots__/**' \
  -g '**/testdata/**' -g '*.{json,yaml,yml,toml,ini,tf,tfvars,env*}' \
  -e 'gpt-|claude-|gemini-|api\.openai\.com|api\.anthropic\.com'

Finally, ask history. A model string that was removed and reintroduced on a branch shows up in the log even when the working tree is clean:

git log --oneline -S 'gpt-4' -- . | head -40
git log --oneline -S 'api.anthropic.com' -- . | head -40

Copies that are not in the repository at all

This is the part of the audit that a search tool cannot do for you, and it is where cutovers fail. A repository grep has no reach into running state. Check each of these by hand:

• Database rows. Tenant settings, feature-flag values, prompt registry entries, agent definitions, saved “assistant” records. A model string stored per customer is invisible to every search you just ran and outlives every deploy. Go looking: select distinct model from llm_calls order by 1; for what has actually been called, and a pattern scan over any free-form settings table for what is configured.
• Queued and scheduled work. Jobs already sitting in a queue, or serialised into a scheduler, carry the model string that was current when they were enqueued. A cutover that drains the queue after the switch will replay the old name against the new credentials. Either drain first or make the worker resolve the model at execution time rather than trusting the payload.
• Secrets and environment configuration. Values set in a secrets manager, a platform dashboard or a CI variable never appear in the repository. Enumerate them from the platform’s own API, not from the .env.example that claims to describe them.
• Infrastructure and deployment code, if it lives in another repository. Terraform variables, Helm values, task definitions and edge-function configuration are a second codebase with the same problem.
• Observability configuration. This one is consistently forgotten. Dashboard panels filter on a model label; alert rules threshold on a metric series that includes the model name in its tags; log-based metrics parse a field that is about to change shape. Nothing errors — the panels simply go flat, and a flat error rate looks like success. Export the dashboard and alert definitions as JSON and grep those files the same way you grepped the source.
• Already-shipped clients. A mobile build or a desktop app in the field has its own copy and cannot be edited. If a released client sends a model name, the cutover has to keep accepting the old one for as long as that build is supported.

Turning hits into an inventory

1. Run each search with --json or plain -n output into one file per category, so the four categories stay separable.
2. Classify every hit into exactly one of: call site (a live request), fixture (a test expectation), fallback (a branch reached only on error), documentation, or dead. The fallback branches are worth marking explicitly — they are the least-tested code in the repository and the code most likely to run during a migration.
3. For each call site, record the model, the token budget applied to it, and whether the surrounding code branches on a provider-specific error or stop value. That triple is the actual migration unit.
4. Count the distinct models. Teams usually expect three and find eleven, most of them one-off choices made during an experiment that was never cleaned up.
5. Cross-check the inventory against reality by comparing the distinct models in your call logs to the distinct models in your source. A model in the logs but not the source is coming from stored configuration; a model in the source but not the logs is dead code you can delete instead of migrate.

Keeping the count from going back up

The audit has a short half-life. Once the inventory is at zero, add the same searches to CI as a failing check restricted to application code, with the configuration module as the single allowed exception. A five-line CI step that greps for vendor prefixes and exits non-zero outside one directory is enough, and it is the difference between doing this once and doing it again next year. The follow-on step — collapsing the inventory into one place — is centralising model configuration.