Building a Fine-Tuning Dataset From Production Logs

Production logs are the best training data you will ever have and the most dangerous. They are real inputs in the real distribution — and they are full of your current model’s outputs, which is exactly what you must not train on.

Selection is the entire job

A log dump is not a dataset. Ninety percent of production traffic is the model already doing the right thing, and training on it teaches nothing you do not already have. The examples worth including are the ones carrying information the model currently lacks.

• Requests where a human edited the output. The highest-signal source in any product that has a review step. The edit itself is the label: input plus the corrected output is a perfect training pair, and the diff tells you which failure mode you are fixing.
• Requests that failed a validator. Schema violations, failed JSON parses, outputs that broke a downstream contract. Pair the input with a hand-written or repaired correct output.
• Requests that got retried. A user regenerating is an unlabelled negative. It tells you where to look even when it does not tell you what was wanted.
• Requests routed to a more expensive model. If a fallback or escalation path exists, everything that took it is a candidate for exactly the thing a fine-tune is good at: teaching a cheap model a behaviour the expensive one has.
• A stratified sample of ordinary traffic. Not because it teaches, but because a dataset made entirely of failures teaches the model that everything is a failure case. Ten to thirty percent ordinary traffic keeps the distribution honest.

Where the correct output comes from

Selection gives you inputs. Targets are a separate problem and there are only three sources, in descending order of quality and cost.

Cleaning without flattening

The instinct is to normalise everything: strip whitespace, standardise casing, unify date formats, template the boilerplate. Resist most of it. Your inputs at inference time will be messy in exactly the ways your logs are messy, and a model trained only on clean inputs handles messy ones worse.

What genuinely needs doing:

• Redact, do not delete. Personal data has to come out, but replacing a name with an empty string changes the shape of the sentence. Replace with a plausible surrogate of the same type and length class, consistently within an example.
• Deduplicate at two thresholds. Exact-match dedup on a hash catches replays. It does not catch the templated request that differs by one identifier, and log data is full of those. Near-dedup with MinHash at a Jaccard threshold around 0.8 does, and on log-derived corpora it typically removes far more than exact dedup.
• Cap per-source contribution. One enterprise customer generating 60% of your traffic will generate 60% of your dataset and you will fine-tune a model for them specifically. Cap any single tenant, template or endpoint at a fixed share.
• Drop the truncated. Any completion that stopped at max_tokens rather than an end-of-sequence token is a training example that teaches the model to stop mid-sentence. Filter on the finish reason, not on length.

The two leakage traps

Near-duplicate leakage across the split

Split by random sampling and the templated request that appears 400 times lands in both train and test. Your evaluation then measures memorisation and reports it as generalisation, and the number is spectacular right up until production disagrees with it.

Deduplicate before splitting, and split on a grouping key rather than on rows — by tenant, by document, by session, by template id. If two examples could plausibly have come from the same underlying thing, they belong on the same side of the split.

Temporal leakage

A random split also lets the model train on next month and test on last month. For anything where the input distribution drifts — and product traffic always drifts — split by time: train on everything before a cutoff, test on everything after. The resulting number is lower and it is the one that predicts production.

The feedback loop nobody notices

This is the trap specific to log-derived data and it is quiet. Your logs contain your current model’s outputs. If you train on them and deploy, the next month of logs contains the new model’s outputs, which you then train on. Each generation is fitted to the previous generation rather than to what users wanted.

The visible symptom is narrowing: outputs become more uniform, hedging disappears, the model develops verbal tics, and rare-but-valid response shapes vanish entirely. It is the same dynamic described for synthetic data in Shumailov et al.’s 2024 Nature paper on recursively generated training data, arriving through the back door of a logging pipeline.

The defence is a hard rule rather than vigilance: an example only enters the dataset if its target came from outside the model — a human edit, a passing test, a reconciled transaction, a different model. Log the provenance as a column, and make the training job refuse rows without one.

Formatting and loss masking

The last step is mechanical and is where a surprising number of runs fail silently. Two rules.

Use the base model’s own chat template. Not one that looks like it. The special tokens, the role markers and the whitespace all matter, and if the template you train with differs from the template your serving stack applies, the adapter is being asked to generalise across a formatting shift for no reason. Render one example and print it with the special tokens visible before you launch a run.

Mask the loss on the prompt. By default many training scripts compute loss over the whole sequence, so the model is being trained to predict your instructions as well as its answer. For instruction tuning you almost always want completion-only loss:

tokens   <|system|> You are ... <|user|> Summarise ... <|assistant|> The report ... <|eot|>
labels   -100 -100 -100 ... -100 -100 -100 ...        The report ... <|eot|>
         \________________ masked ________________/  \____ supervised ____/

The end-of-turn token must be inside the supervised region. Mask it and the model is never trained to stop, which produces the single most common “the fine-tune broke everything” report: a model that answers correctly and then keeps going forever.