How Much Data Do You Need to Fine-Tune?
5 min read · updated August 3, 2026
The honest answer is between 500 and 500,000, and which end you are at depends almost entirely on whether you are teaching behaviour or coverage. Fortunately the question is cheap to answer empirically, and the procedure at the end of this page is worth more than any number.
Why the number is unanswerable as asked
“How many examples” conflates two quantities that behave completely differently. One is how many examples it takes for the model to learn the pattern — the output shape, the register, the decision rule. The other is how many it takes to cover the input space — every document type, every edge case, every locale.
The first number is startlingly small, because the base model already knows how to write JSON and how to be terse; you are selecting an existing behaviour, not installing one. The second number is bounded by the diversity of your inputs and has nothing to do with the model at all. A task with four input shapes needs a fraction of the data of a task with four hundred, at identical difficulty.
Three published anchors
| Dataset | Description |
|---|---|
| LIMA — 1,000 | Zhou et al., 2023 (arXiv 2305.11206). One thousand carefully curated prompt-response pairs, used to instruction-tune a 65B base. The authors' Superficial Alignment Hypothesis: a model's knowledge comes almost entirely from pretraining, and alignment tuning mainly teaches it which format and style to use. |
| Alpaca — 52,002 | Stanford CRFM, March 2023. Generated by self-instruct from an OpenAI model. The release notes put the data generation at roughly $500 of API spend and the 7B training run at under $100 on 8x A100-80GB in about 3 hours — figures from 2023 that are useful for the ratio between data cost and compute cost, not for the absolute prices. |
| Flan — 1,836 tasks | Chung et al., 2022, Scaling Instruction-Finetuned Language Models (arXiv 2210.11416). Scaled the number of distinct tasks rather than examples per task, and reported that task diversity was the axis that mattered. |
Read together these say something more useful than any single figure. LIMA says a thousand examples can be enough when what you want is behaviour. Flan says that when you want breadth, the axis to scale is the number of distinct kinds of example, not the count. Alpaca says that curating fifty thousand examples cost more than training on them — which is still true and still surprises people.
The shape of the curve
For a narrow behavioural target, held-out performance against dataset size is steep and then flat. The first few hundred examples move the metric a great deal, the next few thousand move it a little, and beyond that additional examples of the same kind do essentially nothing — you have already communicated the pattern, and repeating it does not communicate it harder.
The flat region is where two failure modes live. Teams add data, observe no improvement, conclude fine-tuning does not work, and stop. Or they add data, observe no improvement, and train for more epochs instead — which does move the training loss, by memorising, and moves held-out performance the wrong way.
What actually breaks the plateau is a different kind of example, not more of the same kind. If the curve has flattened, go and find the inputs your dataset does not contain rather than doubling the ones it does.
Dataset size and epoch count trade against each other in a way that is easy to get backwards. What the optimiser sees is tokens times epochs, so 2,000 examples for three epochs and 6,000 examples for one epoch present similar amounts of gradient signal — but they are not equivalent, because the first shows the model each example three times and the second shows it three times as many distinct examples. The first memorises, the second generalises. When you are short of data the instinct is to raise epochs to compensate; the arithmetic looks fine and the outcome is worse. Raising epochs is a way to extract more from data you have, not a substitute for data you do not.
A rule of thumb by task type
| What you are teaching | Description |
|---|---|
| Output format / schema | 200–1,000. Format is the easiest thing to teach. If 1,000 clean examples have not fixed schema compliance, the problem is the schema or the data, not the volume. |
| Tone, register, house style | 500–2,000, and consistency matters more than count. Two hundred examples that all agree beat two thousand written by six people who disagree. |
| Classification, fixed label set | 50–200 per label as a floor, more for labels that are rare or confusable. Balance matters; a label with four examples will not be learned. |
| Multi-step task with tool calls | 2,000–10,000. Each distinct trajectory shape needs coverage, and the number of shapes is what drives the count. |
| Preference / behavioural alignment | 5,000–50,000 pairs. Preference signal is noisier per example than supervised signal, so it needs more of it. |
| New factual knowledge | No amount is the right answer. See /learn/format-vs-knowledge-tuning. |
The ablation that answers it properly
This costs a few hours of GPU time and removes the guessing entirely.
- Hold out a test set first, before any subsetting, and deduplicate it against everything else. If you sample the test set after you have already split, near-duplicates leak and every number below is optimistic.
- Train on 25%, 50%, 100% of the remainder with identical hyperparameters and identical seeds. Three runs. On a LoRA setup these are hours, not days.
- Plot held-out performance against training set size. The shape of the last segment is the answer. Still climbing between 50% and 100%: more data of the same kind will help. Flat: more of the same kind will not, and you need different data or a different method.
- Run the same three checkpoints against a general-ability suite, not just your task. If your task metric is flat and your general metric is falling, you are past the point where extra data helps and into the region where it costs.
The 25/50/100 sweep is the single highest-value experiment in this whole cluster, and it is almost never run, because it feels like spending three times the compute to learn something you could have guessed. You could not have guessed it.