Prompt Portability From Frontier to Open Models

A prompt is not portable, and the reason is not that the open model is worse. It is that a prompt tuned against one model has accumulated dependencies on that model’s particular habits, and most of them are invisible until they break.

The assumption that fails

Prompts are written iteratively against a specific model. Every round of “that did not work, let me rephrase” encodes something about how that model interprets instructions — its default verbosity, how much it infers from an example, whether it reasons before answering without being asked, how it resolves a conflict between two instructions.

None of that transfers. So a prompt moved to an open model is being asked to do two jobs at once: describe the task, and rely on conventions the new model never learned. When output quality drops, the honest first question is which of the two is failing — and the answer is more often the second than people expect.

Check the chat template first

Before diagnosing anything about your wording, eliminate the failure that accounts for a surprising share of “this open model is bad” reports. Every instruction-tuned model was trained with a specific token layout for roles, turn boundaries and system messages. Serve it under a different layout and it is being asked to work out-of-distribution.

# print what the model's own template produces for your messages
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("org/model-8b-instruct")
print(tok.apply_chat_template(messages, tokenize=False,
                              add_generation_prompt=True))

# compare against what your serving stack actually sent.
# they must match exactly, including special tokens and whitespace.

Two related traps. Some models do not support a system role at all and their template silently folds it into the first user turn — if your prompt leans on a long system message, that changes its meaning. And a GGUF repackaged by a third party may carry a template that differs from the publisher’s. Check the rendered string, not the intention.

The five failure modes

1. Instruction density

A frontier model can hold a dozen constraints simultaneously. A smaller one holds fewer and drops the rest, usually the ones mentioned earliest or buried mid-paragraph. The symptom is output that satisfies most of the requirements and quietly ignores two.

2. Implicit reasoning

Prompts written for capable models often assume the model works through the problem before answering — either because it was trained to, or because the task simply needed it. A smaller model answers immediately from the first plausible continuation. The symptom is answers that are confidently wrong in a way that a moment of working would have caught.

3. Format compliance

“Respond in JSON” is a soft instruction. Frontier models treat it as near-absolute; smaller ones add a preamble, wrap it in a code fence, or emit almost-valid JSON with a trailing comma. The symptom is a parser failing on a small percentage of requests.

4. Abstraction and negation

Instructions phrased as principles — “be concise but thorough”, “use judgement” — and instructions phrased as prohibitions both transfer badly. Negations are notoriously weak: telling a model not to do something frequently raises the probability it does.

5. Length and stopping

Verbosity defaults differ sharply, and smaller models are more prone to continuing past a natural end, repeating a structure, or restating the question. The symptom is output that is right but three times longer than it should be.

The repair procedure

Apply in order and re-test after each step. Doing them all at once means never knowing which mattered.

• Fix the template. Non-negotiable and free.
• Add two or three examples. The highest-yield single change. Smaller models infer far more from a demonstration than from a description, and examples fix format, length and register at once.
• Ask for reasoning explicitly. If the task needs steps, say so and give the steps names. Where you cannot afford the tokens in the answer, ask for the reasoning inside a delimiter and strip it.
• Constrain the output mechanically. A JSON schema or grammar at sampling time makes format failures impossible rather than rarer. This is the repair for mode three, and it is better than any wording.
• Convert principles into rules and negations into positives. “Be concise” becomes “at most three sentences”. “Do not mention the source” becomes “refer only to the summary”.
• Split the prompt. If it carries four instructions that keep getting dropped, it is two prompts. Chaining two reliable steps beats one unreliable one, and it is cheap on a local model.
• Move constraints to the end. Instructions nearest the generation point are followed most reliably. Restating the critical one immediately before the output is a legitimate trick.
• Set explicit stopping conditions — a length limit, a stop sequence, a required closing token.

Keeping both prompts alive

Once repaired, resist the urge to run the adapted prompt on both models. Prompts and models are a matched pair, and a prompt optimised for a small model is usually over-specified and needlessly long for a large one — you pay for the extra tokens and sometimes get worse output, because a capable model given six explicit steps follows six steps instead of finding a better path.

• Keep one prompt per model in version control, with the model identifier in the filename. Two files is not duplication; it is the honest representation of the situation.
• Keep one evaluation set for both. The task is the same even when the prompt is not, and a shared eval is what makes the comparison meaningful.
• Re-test on model updates. A new version of the open model may no longer need the scaffolding, and carrying repairs forward forever costs tokens and quality.
• Record why each repair exists. A comment saying which failure mode a paragraph is defending against is what lets the next person delete it when it stops being needed.