Exporting a Fine-Tuning Dataset Between Provider Formats

A fine-tuning file is JSONL, and converting between provider shapes looks like a rename. It is not. The old format encoded structure in punctuation — a separator string, a leading space, a stop sequence — and if you carry those into a message array you have trained a model to emit them.

The two shapes

Almost every fine-tuning file you will meet is one of two shapes, one line of JSON per example.

The older prompt/completion shape is a pair of strings, from the era when the underlying endpoint took a single prompt string and continued it:

{"prompt": "Ticket: card declined at checkout\n\n###\n\n", "completion": " billing_issue END"}
{"prompt": "Ticket: how do I export my data\n\n###\n\n", "completion": " account_admin END"}

The messages shape mirrors a chat request: each line is an object containing a messages array with the same roles a request uses, and the assistant turn is the target.

{"messages": [
  {"role": "system", "content": "Classify the support ticket."},
  {"role": "user", "content": "Ticket: card declined at checkout"},
  {"role": "assistant", "content": "billing_issue"}
]}

A third family exists on APIs whose request body is not messages-shaped at all. Google’s Gemini API represents a conversation as contents, an array of turns with a role of user or model and a parts array, with the standing instruction carried separately (Google, generateContent reference). Converting into that family is the same exercise as below with one extra rename and one extra decision about where the system instruction goes.

What does not convert

These are the parts that make the conversion an editorial job rather than a script, and every one of them is a way to train a model to do something you did not intend.

The separator

Prompt/completion datasets conventionally end the prompt with a fixed separator — "\n\n###\n\n" is the one you will see most often — so the model learns where input stops and output begins. A chat template does that job structurally, with role markers the serving stack inserts. Leaving the separator in the user message teaches the model that a string of hashes precedes every answer, and at inference time an input without one is now out of distribution. Strip it, and strip it from your inference-time prompt in the same commit.

The leading space

Completions in the old shape conventionally begin with a single space, because tokenizers usually attach a leading space to the following word and the training text needs to match what the model will see after the prompt. In a message array the assistant turn starts at a role boundary and there is no such continuation. Carry the space across and you have trained every reply to begin with whitespace, which shows up later as strings that fail an exact-match comparison for no visible reason. Trim it.

The stop sequence

The old shape often ends completions with a sentinel — END, \n\n, ### — that you then passed as a stop string at inference so generation halted. Chat models stop on their own end-of-turn token. A sentinel left in the training data becomes literal output the model emits, and if you no longer pass the matching stop string it appears in your users’ text. Strip it from the target and delete the corresponding stop parameter from the call site.

The system message

Prompt/completion has no system role. Whatever standing instruction your inference-time prompt carries was either repeated in every training prompt or was absent from training entirely. Both cases need a decision, not a default. If you add a system message to every converted example, add the same string you will send at inference — a model fine-tuned with one system prompt and served with another has been trained on a distribution it will not see. If you add none, do not send one at inference.

Loss masking

Multi-turn examples raise a question the old shape could not ask: in a conversation with three assistant turns, which are trained on? Some fine-tuning formats accept a per-message weight, conventionally weight: 0 to include a turn as context without training on it. Converting a single-turn dataset you will not need this. Converting a multi-turn dataset from a format that trained only the final turn, you must set it explicitly, because the default is usually to train on all assistant turns and that silently changes what you are teaching.

Tool calls

If the source dataset contains tool invocations, they do not survive a naive conversion at all: the assistant turn’s content is not a string, and the tool result is a separate turn with its own role and its own correlation field. The mapping is the same one described in mapping multi-turn tool-call sequences, and it is where a converter most often silently drops turns.

The converter

Standard library only, streaming line by line so a large file does not have to be resident, and refusing rather than guessing when it meets something it does not understand.

#!/usr/bin/env python3
"""prompt/completion JSONL -> messages JSONL."""
import json, sys, argparse

def convert(row, separator, stop, system):
    prompt, completion = row["prompt"], row["completion"]

    # 1. Separator: structural in the old shape, noise in the new one.
    if separator and prompt.endswith(separator):
        prompt = prompt[: -len(separator)]
    elif separator:
        raise ValueError("prompt does not end with the declared separator")

    # 2. Stop sentinel: the chat template supplies the end of turn.
    if stop and completion.endswith(stop):
        completion = completion[: -len(stop)]

    # 3. Leading space: a tokenizer convention with no counterpart here.
    completion = completion.strip()
    prompt = prompt.strip()

    messages = []
    if system:
        messages.append({"role": "system", "content": system})
    messages.append({"role": "user", "content": prompt})
    messages.append({"role": "assistant", "content": completion})
    return {"messages": messages}

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("infile")
    ap.add_argument("outfile")
    ap.add_argument("--separator", default="\n\n###\n\n")
    ap.add_argument("--stop", default="")
    ap.add_argument("--system", default="",
                    help="must equal the system prompt you will send at inference")
    a = ap.parse_args()

    seen, dupes, bad = set(), 0, 0
    with open(a.infile, encoding="utf-8") as src, \
         open(a.outfile, "w", encoding="utf-8") as dst:
        for n, line in enumerate(src, 1):
            line = line.strip()
            if not line:
                continue
            try:
                out = convert(json.loads(line), a.separator, a.stop, a.system)
            except (ValueError, KeyError) as e:
                print(f"line {n}: {e}", file=sys.stderr)
                bad += 1
                continue
            key = json.dumps(out, sort_keys=True)
            if key in seen:
                dupes += 1
                continue
            seen.add(key)
            dst.write(json.dumps(out, ensure_ascii=False) + "\n")
    print(f"written; {dupes} duplicates dropped, {bad} lines rejected",
          file=sys.stderr)

if __name__ == "__main__":
    main()

The choice worth defending is raising on a prompt that does not end with the declared separator instead of passing it through. A dataset with inconsistent separators is a dataset assembled by more than one process, and the examples that differ are exactly the ones you want to look at by hand. A converter that silently accepts both produces a file that trains fine and behaves inconsistently.

Going the other direction — messages back to prompt/completion — is lossier and mostly ill-advised: you must serialise roles into a single string, invent a separator, invent a stop sentinel, and decide what to do with multi-turn examples. If you have to, flatten with explicit role labels and keep the separator and sentinel in a constant shared with the inference code, because those two strings are now part of the model’s contract.

Validating before you upload

Four checks, cheap to run, and each catches a class of failure that otherwise costs a full training run to discover.

• Every line parses and has the required shape. One malformed line can reject the whole upload, and the error message usually names a line number in a file you generated, not in the source.
• No example exceeds the model’s context window once templated. Count tokens with the target model’s own tokenizer, not an estimate, and remember the chat template adds tokens per turn that a naive count of the content misses. Over-long examples are usually truncated rather than rejected, which quietly removes the assistant turn — the part you are training on.
• No leftover separators or sentinels. Grep the output for the separator string and the stop string. Both should return nothing. This one catches the most common conversion bug in seconds.
• The system prompt matches the call site. Compare the string in the file against the string in the code that will call the fine-tuned model. If they differ, one of them is wrong.

Run it

1. Read the target provider’s current fine-tuning file documentation. Shapes and limits move, and the format is validated on upload.
2. Inspect the first and last lines of the source file to identify the separator and stop sentinel actually in use. Do not assume the conventional ones.
3. Decide the system prompt now, and write it down as a constant that both the converter and the inference call site import.
4. Run the converter. Read the rejected-line count on stderr; if it is not zero, look at those lines before continuing.
5. Run the four validation checks, including a token count with the target model’s tokenizer.
6. Convert twenty lines by hand into the request shape you will use at inference and eyeball them side by side with the file. The mismatch you find here is the one you would otherwise find after paying for a training run.
7. Hold back a validation split before uploading, and keep the source file and the converter in version control together, so the transformation that produced the training data is reproducible.