Migrating a Prompt Engineering Team's Internal Documentation

After a provider migration, part of your internal prompt documentation is wrong and none of it says so. Code that names a dead parameter returns a 400 the first time it runs; a wiki page that names the same parameter costs a colleague an afternoon, repeatedly, for years.

Three kinds of claim in a prompt wiki

Before deciding what to revise, sort what is written. Almost every sentence in a prompt-engineering wiki is one of three things, and they have completely different lifespans.

• Task-bound. True about your problem regardless of which model reads it. “A summary of a claims note must preserve every monetary amount and every date.” “Never let the model choose the currency.” These survive any migration untouched, and they are usually the most valuable pages you have.
• Model-bound. True about a model family’s behaviour. OpenAI’s own GPT-4.1 prompting guide states that the model “is trained to follow instructions more closely and more literally than its predecessors, which tended to more liberally infer intent”, and warns that existing prompts may not immediately carry over. See the GPT-4.1 prompting guide. A page of tips written against a model with the opposite tendency is not merely stale — following it now produces different behaviour than it used to.
• Provider-bound. True about an API surface: parameter names, value ranges, defaults, which field carries the system prompt, what the field holding the reason generation stopped is called, which SSE events arrive during a stream. These are the ones a migration invalidates wholesale.

The classification is not academic. It decides the treatment: task-bound pages are left alone, model-bound pages get a dated provenance line and a re-test, and provider-bound claims stop being prose at all.

Why provider-bound claims rot silently

A wiki page is not executed. That is the whole mechanism, and every remedy below follows from it. Nothing reads the sentence “set max_tokens to cap the reply” and checks whether the field is still called that; nothing notices that a code sample no longer compiles; nothing notices that a model name in a table was retired last quarter. The page keeps rendering, looking exactly as authoritative as it did when it was true.

The second-order failure is worse than the first. Once a team has been burned twice by the wiki, they stop trusting all of it, including the task-bound pages that were never wrong. A documentation set does not degrade gradually in usefulness; it crosses a line past which people ask a colleague instead, and then it is dead weight that still costs review time.

Making the rot loud

The rule is short enough to enforce: if a sentence names an identifier, that sentence is generated or tested. Four ways to honour it, in descending order of how much work they save.

Generate the parameter tables

A table of parameters, defaults and ranges should be emitted by the code that builds requests, or from the provider’s published schema, at docs build time. A generated table cannot disagree with the client, which is the only property that matters. A hand-typed table is a copy, and a copy of a moving thing is a bug with a publication date.

Make every code sample a file that runs

Move each snippet out of the page and into the repository, then include it by reference. Now the snippet is type-checked with everything else, and if it is exercised in CI — against a recorded fixture if you do not want live calls — a provider change breaks the build rather than the reader. This single change converts the most dangerous category of wiki content, working-looking code, into the safest.

One constant module for model names

Every model string in the organisation, including the ones in the documentation, resolves from one module. When a model is retired you edit one file and the docs follow. This also gives you the inventory you need to audit aliases, which is the subject of what happens to an alias when the model behind it is retired.

A provenance line on every page

The cheapest item and the one most often skipped: every page opens with the provider, the model and the date it was written against. It fixes nothing by itself, but it converts an unfalsifiable page into a page a reader can discount, and it makes the sweep below possible at all.

The one-time sweep, in order

1. Grep the whole documentation set for identifier-shaped tokens: snake_case and camelCase words, anything in backticks or a code span, anything matching your model-name pattern. This is your candidate list and it will be longer than anyone expects.
2. Classify each hit as task-, model- or provider-bound. Do this before editing anything — the counts per class tell you whether this is an afternoon or a fortnight.
3. For provider-bound hits, apply the executable rule above. If a claim cannot be generated or tested, it is a claim you should not be making in a durable page; move it to a dated note.
4. For model-bound hits, add the provenance line and queue a re-test. Do not edit the advice on the basis of what you assume the new model does; the entire class exists because that assumption is what went wrong last time.
5. Datestamp everything you touched, and record what you did not touch. An untouched page with an old provenance line is honest. An untouched page with today’s date on it is a lie you told yourself.

One thing not to do: a find-and-replace of the old provider’s name. It produces sentences that are grammatically perfect and factually inverted — the parameter semantics stay, the vendor name changes, and the result is more convincing than the original error. Every replacement should be a decision, and if there are too many to decide individually, that is the signal to delete the page.

Deletion is the migration’s one privilege

Normally nobody is allowed to delete internal documentation, because somebody might need it. A migration suspends that, briefly, and it is worth spending. Any page whose owner cannot be named, whose provenance predates the previous provider, and which nobody has opened in a year, goes. Keeping it is not free: it dilutes search, it gets cited in reviews, and it occupies the same shelf as the pages that are correct.

What replaces the deleted pages is usually not a page. Terminology belongs in a glossary that migrates deliberately, the prompts themselves belong in a registry rather than in prose, and the process knowledge belongs in the gate described in migrating a prompt approval workflow. The wiki keeps what is genuinely explanatory, which is far less than it currently holds and far more useful.