What to Pull From a Migration Target's Model Card, and What to Update

The model card is the only document the vendor publishes that is intended to be read before you deploy, and in most migrations it is read once, by one person, who quotes the context window in a Slack thread and closes the tab. There are seven fields in it that invalidate something you have already written down.

What a model card is for

The format comes from Mitchell et al., “Model Cards for Model Reporting” (2019), whose proposal was that a released model should ship with a short structured document covering intended use, out-of-scope use, evaluation conditions, and known limitations — so that a deployer can tell whether their situation is one the model was built and tested for. The general treatment is in model cards; this page is about the specific act of reading one during a migration.

The migration framing matters because you are not evaluating a model in the abstract. You already have a system with documented assumptions in it, and every one of those assumptions was derived from a previous model’s card or from behaviour you observed. The job is to find which assumptions the new card contradicts.

The fields worth pulling

• Training data cutoff. The date after which the model has no parametric knowledge. Usually stated plainly; occasionally stated as a range or with a caveat about later data in some domains.
• Intended use and out-of-scope use. The section people skip. It is the vendor telling you which deployments they will and will not stand behind, and it is the section your legal and compliance reviewers will ask about later.
• Known limitations. Enumerated weaknesses — often around long-context recall, arithmetic, low-resource languages, or specific reasoning failure modes. These are hypotheses to test, not facts to accept, but they tell you where to point your fixtures.
• Evaluation conditions. Which benchmarks, at which settings, with which prompting. A benchmark score reported with an elaborate scaffold does not predict your zero-shot behaviour, and the card is where that is disclosed.
• Supported modalities, languages and context length. Including the distinction between what is accepted and what is evaluated. A model that accepts a million tokens and was evaluated at a fraction of that is telling you something.
• Safety and refusal behaviour. Which categories are refused, and whether the refusal is surfaced as an ordinary response or as a distinct signal your code can branch on. This one changes error handling.
• Deprecation and retirement dates. Sometimes on the card, more often on a separate lifecycle page. Either way it belongs in your extraction, because it sets the clock on the next migration.

What each field makes stale

The point of the extraction is the second column. Each field, and the internal artifact it invalidates:

| card field              | internal artifact to update                    |
|-------------------------|------------------------------------------------|
| training cutoff         | RAG freshness policy; the "the model knows      |
|                         | about X" assumptions in prompts; any prompt     |
|                         | that states today's date relative to it         |
| intended / out-of-scope | vendor review record; DPIA or equivalent;       |
|                         | the "approved uses" list in your AI policy      |
| known limitations       | the edge-case suite (targets for new fixtures); |
|                         | the model-selection decision tree               |
| evaluation conditions   | any internal benchmark comparison that quoted   |
|                         | the vendor's number without the conditions      |
| modalities / languages  | capability matrix; per-locale rollout plan      |
| context length          | truncation strategy; compaction thresholds      |
| refusal behaviour       | on-call runbook; the error-handling branch that |
|                         | distinguishes refusal from failure              |
| retirement date         | the migration calendar; contract renewal review |

The training cutoff row is the one that most often produces a real bug rather than a stale document. A retrieval system tuned on the assumption that the model knows nothing after a given date will over-retrieve or under-retrieve against a model with a different one, and the symptom is subtle: answers that are correct but redundantly hedged, or answers that confidently use stale parametric knowledge because your retrieval did not think to cover that period. The consequences are worked through in knowledge cutoff differences across a migration.

The capability matrix is the artifact most likely to exist and most likely to be wrong, because it is usually a table someone built during the last selection exercise and never revisited. Updating it from the card is an hour of work that prevents a category of decision made on a stale row — see migrating a capability matrix.

What a card cannot tell you

Being clear about this is what stops the card from being used as evidence it cannot support:

• How it behaves on your prompts. The card describes the model; your system is the model plus a prompt plus a schema plus a retry policy. Nothing in the card predicts the combination.
• Latency and throughput under your load. These are properties of the serving infrastructure and your account’s limits, not of the model, and they are not on the card.
• Where exactly the refusal boundary sits. The card names categories. Whether your specific security-research or medical-summarisation inputs land inside them is an empirical question with an answer that changes between versions.
• Tokenizer behaviour on your text. A card may mention the tokenizer family. It will not tell you what your Thai product catalogue costs, which is a question with a counting endpoint for an answer.
• What changed since the last version. Cards describe a model, not a diff. The absence of a change from the previous card’s text is not evidence that nothing changed.

Recording it so it can be re-run

The extraction has a short shelf life. Cards are edited in place: figures get corrected, limitations get added after launch, evaluation sections get expanded, and none of that reliably comes with a version number or a changelog entry. An extraction with no date on it is unfalsifiable a quarter later.

The failure mode this produces is specific and common: a team quotes a context length or a cutoff in a design document, the card is revised, and the design document goes on being cited for another year by people who reasonably assume someone checked. Nobody lied and nobody is at fault; the document simply had no date on it, so there was never a moment at which it became visibly stale.

So record four things next to every extracted value: the value, the source URL, the date you read it, and the internal artifact you updated as a result. Keep it in the repository next to the prompts rather than in a wiki, so it moves through review with the code that depends on it. Then set the re-read trigger to the two events that actually matter — a new model version in your roster, and a change to your own approved-use list — rather than to a calendar date nobody honours.