Model Cards and Datasheets as Regulatory Evidence

A vendor sends a model card in response to a due diligence questionnaire. The useful question is not whether the card is good, but which of your legal requirements it can actually be filed against — because the answer is fewer of them than the length of the document suggests.

Where the artefacts came from

Neither artefact was designed as a compliance document. Model cards come from Mitchell and colleagues, “Model Cards for Model Reporting” (2018), a research proposal for reporting evaluated performance disaggregated across groups. Datasheets come from Gebru and colleagues, “Datasheets for Datasets” (2018), which borrowed the electronic component datasheet as a metaphor: motivation, composition, collection process, preprocessing, uses, distribution, maintenance.

That origin explains the mismatch you feel reading one during procurement. Both were written to improve transparency between researchers. Neither has a defined scope, a version discipline, a signature, or a required field — and all four of those are what turn a document into evidence.

Fields that map to a requirement

These are the fields that correspond to something a regulator can point at.

• Intended use / out-of-scope use maps directly onto the AI Act’s concept of intended purpose, which appears in the Article 3 definitions and is the first item required in the instructions for use under Article 13(3)(b)(i). It is also the field that determines whether your deployment triggers Article 25(1)(c) by using a non-high-risk system for a high-risk purpose. Of every field on a card, this is the one with the most legal weight.
• Evaluation results with named metrics map onto Article 13(3)(b)(ii), which requires the level of accuracy including its metrics, and onto Article 15, which requires declared levels of accuracy to be stated in the instructions for use. A card reporting a single aggregate score without naming the metric or the evaluation set does not reach this bar.
• Disaggregated performance across groups is the original point of a model card and it maps onto Article 13(3)(b)(v)’s requirement to describe performance regarding specific persons or groups on which the system is intended to be used.
• Limitations and known failure modes map onto Article 13(3)(b)(iii): known or foreseeable circumstances that may lead to risks to health, safety or fundamental rights. This is one of the rare places where a candid research-style limitations section is close to what the law asks for.
• Training data description, for a general-purpose model, maps onto the Article 53 documentation duties and onto the sufficiently detailed public summary of training content required under Article 53(1)(d), for which the AI Office publishes a template. Annex XII, the information a GPAI provider gives downstream providers, is in effect a legally mandated model card and is the right document to ask for by name. See the Article 53 documentation page.

Fields that are supporting evidence

A second tier of fields cannot discharge an obligation but can support an assessment you are required to make.

The datasheet’s collection process and composition sections feed a legitimate interests assessment or a lawfulness analysis for training, and they feed the necessity and proportionality limb of a DPIA under Article 35(7)(b) GDPR. They do not establish lawfulness; they give you facts to reason about. The preprocessing section is where you look for whether personal data was removed, and how — and the answer is almost always weaker than the phrasing suggests, which is itself the finding.

The maintenance and versioning sections support the AI Act’s Article 13(3)(c) disclosure of predetermined changes, and your own change management. The uses section of a datasheet supports a purpose limitation analysis under Article 5(1)(b) GDPR where a dataset is being reused for something other than its collection purpose.

What a card is never evidence of

Three misuses are common enough to name.

A model card is not a record of processing activities. Article 30 GDPR requires a controller to record its own processing: purposes, categories of data subjects and personal data, categories of recipients including those in third countries, transfers, envisaged erasure time limits where possible, and a general description of security measures. Every one of those is a fact about your processing. A vendor’s description of their training corpus does not populate a single field of it. What the vendor’s documentation legitimately feeds is the recipients and transfers entries — and for that you need the sub-processor list, not the model card. Text: Regulation (EU) 2016/679 on EUR-Lex.

A model card is not the Article 11 technical documentation. Annex IV specifies what that documentation contains, it is the provider’s obligation, and it is prepared before the system is placed on the market and kept for ten years. A public card is a summary written for a different audience. See the Article 11 page.

A model card is not a declaration of conformity. Article 47 and Annex V define that document and it is signed. A card asserting that a model “complies with the EU AI Act” is a marketing statement, and treating it as an assurance is the single most common error in this area.

Making a card usable as evidence

The gap between a card and evidence is mostly about fixity, and you can close it from your side without the vendor’s help.

• Archive the card at the version you rely on, with a retrieval date and a hash, in your own storage. Cards are edited in place. The version that described the model you deployed may not exist on the web six months later, and “the vendor said so” without a copy is not a record.
• Bind it to a model identifier you can prove you called. A card describes a named checkpoint; your logs record a model string. If those two cannot be joined, the card describes a model you cannot show you used.
• Record what the card does not say. A due diligence file that notes “no disaggregated performance data published; requested 12 March; vendor declined” is stronger evidence of diligence than a file that silently contains nothing on the point.
• Ask for the Annex XII information by name where the supplier is a general-purpose model provider. It is a defined set with a legal basis, which a card is not.