Auditing a Prompt Library for Provider-Specific Idioms

Before a migration, somebody has to read every prompt in the system and decide whether it is talking to a model or to one particular model. Done as a vibe check it takes a week and misses the important ones. Done as a search pass against a fixed list of categories it takes an afternoon and produces a queue.

Get an inventory first

You cannot audit what you cannot enumerate, and in most systems the prompts are not all in one place. They are in a prompt registry, in template files, in string literals inside service code, in database rows for tenant-specific overrides, and in test fixtures that will start failing for reasons nobody connects to the migration.

1. Pull everything into one flat list: an identifier, the full text, the call sites that use it, and the request parameters normally sent alongside. The parameters are not optional context — one whole category below is invisible without them.
2. Add a usage count from your logs. Ninety per cent of a prompt library is usually dormant, and you should know which ten per cent carries production traffic before you spend effort anywhere.
3. Record the model each prompt was written against, if you can recover it. A prompt written for the model you are leaving is a different risk from one that has already survived a swap.

If this step is hard, that is itself a finding. Prompts scattered across a codebase are prompts that get audited once and then drift; a prompt registry exists to make the next migration a query instead of a search.

The nine things to flag

Visible in the text

• 1. Delimiter conventions. XML-style tags, markdown heading levels, triple quotes, fenced blocks, custom markers like --- or ===. Not wrong, but tuned. Flag every prompt whose structure depends on one, and note which convention.
• 2. Vendor and product names in the instruction. “You are ChatGPT”, “as an OpenAI model”, “use your built-in browsing”. These become false statements after the swap, and a model asked to roleplay as a different vendor’s product behaves unpredictably. Also flag references to capabilities the target may not have.
• 3. Literal turn markers inside a single message. Prompts that embed Human: and Assistant: as text, usually a survivor of a completion-era API. On a chat API these are content, not structure, and on some models they actively confuse the turn boundary.
• 4. Prose that is really a parameter. “Respond deterministically”, “use no more than 500 tokens”, “stop after the closing tag”. A model cannot set its own temperature and cannot count its own tokens. These worked, when they worked, because someone also set the parameter. Flag them and check the call site.
• 5. Meta-instructions about reasoning. “Think step by step before answering”, “show your working in <scratchpad> tags”. Against a reasoning-tuned target these can duplicate or fight the model’s own process and inflate cost for no gain.

Visible only with the call site

• 6. Format compliance borrowed from a parameter. The biggest one. A prompt saying “reply with JSON only” that has a 100% valid-JSON rate is usually not a good prompt; it is a prompt sent with a constrained-decoding option enabled. Move it to a provider without that feature, or with a different schema dialect, and the prompt alone will not hold. Flag any prompt whose call site sets a response-format or schema option, and treat the prompt as unproven until re-tested bare.
• 7. Stop-sequence dependence. A prompt that produces clean output because a stop sequence truncates the model’s trailing chatter. Stop sequences have different limits and semantics per provider, and the truncated remainder reappears the moment one is dropped.
• 8. Assistant-turn prefill. Some APIs let you seed the start of the assistant’s reply to force a shape; others reject a trailing assistant message outright. A prompt that relies on prefill is not a prompt, it is a prompt plus a message-list trick, and the trick may not exist on the target.
• 9. Few-shot examples carried as separate turns. Whether your examples are alternating message turns or one pasted block changes their weight. Record which, because the conversion is covered separately in migrating few-shot examples.

The search pass

Run these over the exported inventory, not over the repository, so database-stored prompts are included.

# 1 delimiters
rg -n '(<[a-z_]+>|^#{1,3} |"""|```|^---$|^===)' prompts.txt

# 2 vendor and product names, and capability claims
rg -ni '(chatgpt|gpt-[0-9]|openai|claude|anthropic|gemini|llama|mistral|as an ai (model|assistant) (made|developed|created) by)' prompts.txt
rg -ni '(your (built-?in|native) (browsing|search|code interpreter|vision))' prompts.txt

# 3 literal turn markers
rg -n '^(Human|Assistant|System|AI):' prompts.txt

# 4 prose standing in for a parameter
rg -ni '(temperature|deterministic|no more than [0-9]+ tokens|token limit|stop (after|when you reach))' prompts.txt

# 5 reasoning meta-instructions
rg -ni "(think step[- ]by[- ]step|let'?s think|show your (work|reasoning)|scratchpad|chain of thought)" prompts.txt

# 6-8 must be joined against call sites, not text
rg -n '(response_format|json_schema|json_object|strict *: *true|stop *[:=]|stop_sequences)' src/

For 6 through 8 the input is your service code, and the output is a list of prompt identifiers. Join it back onto the inventory. In most audits this join finds the prompts nobody flagged by eye and the ones that break first.

Triage by severity

Not every hit is work. Sort into three buckets and only the first two block the migration.

• Blocking — the prompt cannot function as written on the target. A named capability that does not exist; a prefill the target rejects; a schema option with no counterpart. These need a rewrite before any traffic moves.
• Degrading — the prompt will run and produce worse output. Delimiter mismatch, borrowed format compliance, reasoning instructions against a reasoning model, verbosity tuned to the old model. These need a measured re-test, and most of them turn out to need a small edit.
• Cosmetic — a vendor name in a comment, a dormant prompt with no traffic, a test fixture nobody reads. Record and move on. An audit that treats these as work does not get finished.

Give every blocking and degrading item an owner and a target date in the same table as the inventory. The output of this exercise is a queue, not a document.

Verifying the rewrite

1. For each rewritten prompt, keep both versions and version them explicitly. You will want to run the old one against the new model to confirm the rewrite was necessary, and the new one against the old model to confirm you have not created a rollback hazard.
2. Assemble a small fixed input set per prompt out of production traffic — twenty to fifty real inputs is enough to see a structural change — and store the outputs of all four combinations.
3. Assert on structure and content, not on strings. Valid JSON, required fields present, length inside the band, no forbidden phrase. Exact output equality across models is not achievable and is not the goal, which is the point of testing prompt portability across models.
4. Re-run the whole pass as a pre-migration gate rather than as a one-off. The prompts will keep changing while the migration is in flight, and a prompt written next week against the old model is a new instance of the same problem.