What Changes in Your Prompts When You Move From Completion to Chat

The call-site rewrite is an hour. The prompt rewrite is the part that changes your output quality, and doing it as a mechanical wrap — the old string dropped into one user message — keeps every habit the old format required and every one of them now costs you something.

What actually changed underneath

A completion model was handed exactly the bytes you sent and continued them. Your prompt was the context, in full, with no framing added and none removed. That is why the old craft was all about punctuation: you invented separators, you ended with a cue, you passed stop strings, because the model had no other way to know where your input ended and its output should begin.

A chat model is handed a rendered template. Your message array is turned into a single token sequence by a chat template that inserts role markers and turn boundaries the model was trained on, and the model then continues from the start of an assistant turn. Two consequences follow and they drive everything on this page. The boundary work is already done, so doing it again in your text is redundant at best and distribution-shifting at worst. And the role labels are not decoration: the model was trained with instructions arriving in one position and content in another, so the same words move differently depending on which turn they are in.

That last point is the one worth sitting with. Putting your standing instruction in the system position and the variable content in the user position is not organisation for your benefit. It is telling the model which text is policy and which is data — a distinction the completion format could not express at all, which is one reason old prompts were so easy to derail with content that read like instructions.

One prompt, rewritten

Here is a typical completions-era prompt for a classification task, of the kind that has been running unchanged for two years.

You are a support ticket classifier. Classify each ticket into exactly one
of: billing_issue, account_admin, bug_report, feature_request.

Ticket: My card was declined at checkout.
Category: billing_issue

Ticket: How do I change the email on my account?
Category: account_admin

Ticket: {{ticket_text}}
Category:

And with stop set to ["\n"], because otherwise the model would happily invent a fourth ticket and classify that one too. Rewritten, it is this:

{
  "messages": [
    { "role": "system",
      "content": "Classify each support ticket into exactly one of: billing_issue, account_admin, bug_report, feature_request. Reply with the category name only." },

    { "role": "user",      "content": "My card was declined at checkout." },
    { "role": "assistant", "content": "billing_issue" },

    { "role": "user",      "content": "How do I change the email on my account?" },
    { "role": "assistant", "content": "account_admin" },

    { "role": "user",      "content": "{{ticket_text}}" }
  ]
}

Note what happened to the label. The old prompt needed both the Ticket: and Category: prefixes because they were the only structure available. In the message array the roles carry that structure, so the prefixes are gone from the content. The instruction “reply with the category name only” was added, because the trailing Category: cue that used to enforce brevity has been deleted and something has to do that job.

Few-shot examples become turns

The single most common mechanical error in this migration is pasting the few-shot block into the system message as one lump of text. It works. It also throws away most of what few-shot examples do on a chat model.

When each example is a real user turn followed by a real assistant turn, the model sees examples of the exact structure it is about to produce: an assistant turn, in this conversation, containing that kind of content. The template renders them identically to the turn it is being asked to write. When they are prose inside the system message, they are a description of past behaviour rather than an instance of it, and the model has to generalise from the description.

Two practical points follow. First, the examples must be internally consistent in a way a text blob does not require — if one assistant turn is a bare label and another is a sentence, you have shown two formats and will get both. Second, the alternation must be strict. Some APIs reject a message array with two consecutive user turns, and where they are accepted the template may merge them in a way that undoes the structure you were trying to create. If you need multiple inputs in a turn, concatenate them into one message rather than sending two.

Keep the examples in one array in code rather than in a template string. Once they are turns, they are data, and being able to add, remove or reorder them — and to select examples per request — is a real capability the old format made awkward. The general treatment is in few-shot prompting.

Four things to delete

• The trailing cue. Category:, Answer:, Summary: at the end of the prompt. On a completion model this was essential. On a chat model the assistant turn has already started, so the cue is trailing user text, and the common failure is that the model echoes it — you get Category: billing_issue and your exact-match parser breaks. Delete the cue; put the format requirement in the instruction.
• Invented separators. Rows of hashes, dashes, ---, ###, XML-ish fences you added purely to delimit input from output. The turn boundary does that now. Separators that structure content within a message are a different matter and are still useful — marking where a retrieved document starts and ends inside a user turn is good practice, not legacy.
• The stop sequence that ended a turn. A newline stop existed to stop the model inventing the next example. Chat models emit an end-of-turn token by themselves. Worse, that newline is now an active hazard: any legitimate multi-line answer is truncated at its first line break, and the response comes back with a finish reason of stop, so nothing looks wrong. Delete it. Keep only stop strings that are about your content.
• Second-person framing of the model’s own turn. “The assistant then writes…” and similar narration of what is about to happen. It was a way of steering a continuation. Now it is text in the conversation describing the conversation, which reliably produces output that talks about the answer instead of being the answer.

The tricks with no counterpart

Three completion-era capabilities do not exist in a message array, and if your prompt used them you need a plan rather than a rename.

Insertion. The old endpoint accepted a suffix as well as a prompt, so the model filled the middle — the basis of most code-insertion features. A chat request has no notion of text after the generation point. The approximation is to put the following text in the user message and ask for only the middle, which is a different and weaker thing: the model can see the suffix but is no longer constrained to join to it. Where fill-in-the-middle matters, look for a model served with an explicit infilling interface rather than shimming it.

Echo. Returning the prompt along with the completion, which was mostly used together with log probabilities to score existing text rather than generate new text. There is no chat equivalent, and if scoring is what you were doing, the chat endpoint is not the tool.

Continuing a partial answer. Occasionally a completion prompt ended mid-sentence deliberately, to force the shape of the reply. The nearest equivalent is a partial assistant turn at the end of the message array — assistant prefill — which some chat APIs support and others reject outright. It is a genuine parity gap, covered in feature parity gaps and how to shim them. Where it is unavailable, an explicit instruction plus a validator on the first characters of the reply is the fallback.

Do the rewrite

1. Pull the old prompt into three piles: the standing instruction, the few-shot examples, and the variable content. If a sentence does not fall into one, it is probably framing you are about to delete.
2. Put the standing instruction in the system position — a system role message, a top-level system parameter, or a systemInstruction field, depending on the API. Add to it any format requirement that the trailing cue used to enforce.
3. Convert each few-shot pair into a user turn and an assistant turn. Strip the Input: and Output: style labels; the roles have replaced them. Make the assistant turns identical in format to each other.
4. Put the variable content in the final user turn, alone, with no trailing cue.
5. Delete every stop sequence that existed to end a turn or separate an example. Keep only content-driven ones.
6. Run both prompts over the same fifty inputs and diff the outputs by hand. Look specifically for echoed cues, leftover separators in the output, and answers that are longer than the old ones — the three signatures of an incomplete rewrite.
7. Freeze the result as a versioned template. This is the moment to start versioning prompts if you were not, because you now have two variants of the same prompt and will want to know which produced a given logged output.