System Prompts: What Belongs There and What Does Not

Most advice about system prompts assumes the system role is a stronger channel — that the model obeys it harder. That is not a property you can rely on across families. Two other differences are real, and they are enough to decide where every line of your prompt goes.

What actually differs between the roles

Under the hood, roles are text. A chat model is trained on a template that wraps each turn in marker tokens, and the system turn is one more section of the same token stream. There is no separate privileged channel into the weights. What differs is:

• Position. The system prompt is first, so it is the stable prefix of every request in the conversation. That makes it the natural cache boundary.
• Training distribution. Post-training teaches the model that system-turn text is the operator speaking and user-turn text is a member of the public, so a conflict between them tends to resolve toward the system turn. Tends to. It is a prior, not an access control, and it is why prompt injection remains a live problem.
• Variability. By convention the system prompt is constant across requests and the user turn is not. This is a convention you enforce, and it is the one that saves money.

Everything else people attribute to the system role — that it is “weighted higher”, that instructions there are obeyed more literally — varies by family and by version, and designing around it produces prompts that break when you port them.

The hierarchy point is routinely over-read, so it is worth one more sentence. Vendors have published instruction hierarchies — platform above developer, developer above user, user above tool output — and models are trained toward them. That training is real and it is probabilistic. It lowers the rate at which a user turn overrides a system rule; it does not make the rule unreachable, and no model card claims it does. Anything whose violation you cannot tolerate needs enforcement outside the prompt entirely.

The placement rule

One question decides it: would this text be identical on the next request? If yes, it is system. If it changes with the request, it is user — even when it reads like an instruction.

The rule looks trivial and is not, because the tempting mistakes both violate it. Interpolating the customer’s name, today’s date or the retrieved documents into the system prompt makes the prefix unique per request, which silently destroys prompt caching and costs real money at volume. Putting the enduring rules in the user turn buries them under whatever the user typed and makes them injectable.

The awkward middle: things that change rarely

Per-tenant configuration is the case the rule does not answer cleanly. It is constant for a tenant and variable across tenants, so it is neither. Resolve it with volume. Caches are keyed on an exact prefix, so a tenant sending thousands of calls an hour keeps its own prefix warm and should have its configuration in the system prompt; a tenant sending four calls a day never warms anything, and the same text is better placed below the shared prefix, where it costs full price on a handful of requests instead of denying the common block to everybody.

Order the system prompt so that this is possible at all: the block every tenant shares first, tenant-specific text after it, request-specific text last. A cache match truncates at the first difference, so whatever sits above your first varying block is the only part that can ever be reused.

Where each kind of instruction goes

The cache boundary is the real constraint

Providers that offer prompt caching key it on an exact prefix match. The mechanism is described in provider documentation in roughly the same terms across vendors: a cached prefix is charged at a reduced rate on read, sometimes at a premium on write, and any difference in the tokens before the divergence point means no hit at all.

So one interpolated timestamp at the top of a 4,000-token system prompt does not cost you a few tokens. It costs the whole cache. Arithmetic, with your own provider’s numbers substituted for the illustrative ones: 4,000 system tokens at $3 per million input is $0.012 a call, or $12,000 per million calls. At a cache read rate of one tenth, the same traffic is $1,200. The difference between those two numbers is one line in the wrong place.

Which gives the discipline: build the system prompt as a constant, and treat any ${...} inside it as a bug to justify. Everything variable moves down.

Four system prompts that need splitting

• The one with the user’s name in it. Personalisation is a user-turn concern. It is also the single most common cache-buster.
• The one that grew to 8,000 tokens of edge cases. Every incident added a rule and none were removed. Instruction compliance degrades with rule count, so the twentieth rule is quietly weakening the first nineteen.
• The one containing the retrieved context. A prompt whose prefix changes on every request is not a system prompt, it is a template with an expensive habit.
• The one with no output contract. If the parser downstream expects JSON and the system prompt never says so, the format is being decided by whichever example the user happened to include.

The repair for all four is the same and it is not a rewrite. Split the prompt into a constant part and a rendered part, put the boundary at the first variable, and watch two numbers afterwards: the cache hit rate, which should approach one for steady traffic, and the rule count, which should go down at least once a quarter — otherwise you are only ever adding.