Instruction Following: Why Models Ignore Rule 7 of 12

The complaint is always the same: the model followed eleven of the twelve rules. It is treated as a mystery about attention. Most of it is arithmetic, and the arithmetic tells you what to do about it.

The arithmetic of twelve rules

Suppose each rule in your system prompt is obeyed 97% of the time — which for a good model on a simple, verifiable rule is a realistic starting assumption rather than a measurement, and a generous one. What you care about is that every rule holds in a single output:

per-rule    3 rules   6 rules   12 rules   20 rules
   0.99       0.970     0.941      0.886      0.818
   0.97       0.913     0.833      0.694      0.544
   0.95       0.857     0.735      0.540      0.358
   0.90       0.729     0.531      0.282      0.122

P(all rules hold) = p^n , assuming independence

At 97% per rule, twelve rules give you a fully compliant output about 69% of the time. Nothing is broken; three high numbers multiplied twelve times is a mediocre number. Independence is an assumption and a shaky one — rules that conflict fail together, related rules correlate — but the qualitative conclusion survives any reasonable correlation structure: rule count is a budget, and every rule you add taxes the ones already there.

Why compliance is soft

There is no constraint solver anywhere in the system. Each rule is text that shifts a distribution, and the shift has to survive several hundred sequential sampling steps against everything else in the context. That produces three characteristic patterns:

• Later output drifts. Compliance decays with output length: the formatting rule that holds for the first three list items relaxes by the tenth, because the immediate context is now mostly the model’s own prose.
• Conflicting rules resolve silently. “Be concise” and “always explain your reasoning” cannot both be maximised, and the model picks without telling you. Grep your prompt for pairs like this before blaming the model.
• Specific beats general. A concrete rule (“at most three sentences”) survives competition better than an abstract one (“be appropriately brief”), because it constrains the actual token sequence rather than a quality.

A fourth pattern is worth naming because it is diagnosable at review time: rules stated as properties of the output survive better than rules stated as properties of the process. “The reply contains no more than three sentences” describes the artefact and can be checked against the text as it is written. “Think carefully before summarising” describes an activity the model has no way to verify it performed, and no way to be caught not performing.

Measuring it on your own rules

The instrument already exists and it is worth copying rather than citing. Zhou et al. (2023) built IFEval — around five hundred prompts carrying roughly twenty-five types of verifiable instruction: write more than N words, use no commas, wrap the answer in double quotes, mention a keyword at least three times. The insight is not the benchmark, it is the criterion. An instruction that a short function can check is an instruction whose compliance rate you can measure automatically, on every model, forever.

Apply it to your prompt in three steps:

• Write each rule as a checker function returning a boolean over the output. Rules you cannot check are rules you cannot manage — which is itself the finding.
• Run your eval set and record per-rule compliance, not just the all-rules-hold rate. This is what tells you it really is rule 7, and not rules 3 and 9.
• Re-run with the rule set halved. If overall compliance rises, you have confirmed the budget is real for your prompt, on your model.

Expect that per-rule table to be lopsided rather than uniform. In practice a handful of rules account for most of the violations — usually the vague ones, the ones that quietly conflict with another rule, and the ones that apply late in a long output — and that is exactly the information an aggregate compliance number destroys. It also tells you which rule to promote into the schema first, which is the cheapest fix available.

Position, and what is actually known

The common claim is that rules in the middle of a prompt are followed less. The evidence usually cited for it is Liu et al. (2023), Lost in the Middle, which found a U-shaped curve when models retrieve information from long contexts: material at the start and end is used more reliably than material in the middle.

That is a result about retrieving facts, not about following instructions, and stretching it to rules is an inference rather than a measurement. Nobody here has measured the instruction version. The inference is cheap to act on and cheap to be wrong about — put the rules that must not fail at the beginning and immediately before the model starts writing — so act on it, but do not repeat it as an established number.

What to do instead of rule 13

• Move rules into the schema. A rule expressed as an enum or a maxLength in a response schema is enforced by the decoder rather than requested in prose. This removes it from the budget entirely.
• Move rules into validators. Check after generation and retry with the specific violation quoted. The retry works because the violated rule is now the most recent and most specific thing in the context.
• Split the call. Six rules about extraction and six about tone are two prompts of six, which by the table above is 0.83 · 0.83 rather than 0.69 — plus each half is separately testable.
• Delete on evidence. Ablate. Rules added after an incident two years ago are still consuming the budget, and some of them are now inert.
• Merge and concretise. Three vague style rules usually collapse into one checkable sentence, which is one budget item instead of three.

Underneath all five is one reframing. The prompt is not where constraints are enforced; it is where they are communicated. Enforcement belongs to the decoder, the validator, and the code that consumes the output. Prompts written on that understanding shrink over time. Prompts written without it grow one incident at a time until rule 7 of 12 goes missing and nobody can say when it started.