Small Language Models: What 1–8B Can Actually Do

A small model is not a bad large model. It is a different tool, and the tasks it does well are not a scaled-down version of the tasks a large model does well — they are a distinct shape, and once you can recognise the shape the selection becomes easy.

Why small models got good

Models in this range improved faster than model scaling alone would predict, for reasons worth knowing because they suggest the trend continues. Training runs now use far more tokens than the older compute-optimal recipes suggested, so a small model is trained much longer relative to its size. Data curation improved markedly, and quality beats quantity at small scale. And distillation from larger models transfers behaviour that a small model would not learn from raw text.

The result is that capability at a given size has moved substantially while the size itself has not. A model that fits in a few gigabytes does things today that needed a much larger one not long ago — which is exactly why a task list, rather than a model list, is the useful artefact.

What predicts success

Three properties, and they predict outcomes far better than the parameter count:

• Is the information in the context? Tasks where everything needed is in the prompt ask the model to transform rather than to recall. Small models transform nearly as well as large ones and recall much worse.
• How long is the dependency chain? One-step tasks are safe. Each dependent step multiplies the chance of an error, and small models start from a slightly higher per-step error rate, so the difference compounds.
• How constrained is the output? A label from a fixed set is nearly always achievable. Free-form output over a large space is where quality differences become obvious to a reader.

Score your task on those three before choosing anything. All three favourable and a 3B model will probably do; all three unfavourable and no amount of prompt work will rescue a small one.

Where they are sufficient

• Classification and routing. Intent detection, sentiment, triage, spam, language identification, deciding which downstream model or tool handles a request. Fixed output space, everything in context.
• Structured extraction. Pulling fields from invoices, emails, logs or transcripts into a schema. Constrained decoding pushes this from good to reliable.
• Rewriting and reformatting. Tone changes, simplification, converting prose to a template, markdown to JSON. Bounded transformations with the source present.
• Summarising bounded documents. A page, a thread, a meeting transcript. Quality is respectable; it degrades on very long inputs where selection matters more than compression.
• Query rewriting inside a retrieval pipeline. Expanding, disambiguating and generating alternative phrasings — a short, cheap, high-volume step that is nearly ideal for a small model.
• Autocomplete and short code snippets in a familiar language, with the surrounding file in context.
• Speculative decoding. A small model drafts tokens that a large one verifies in parallel, which speeds up the large model with no quality cost. Here the small model’s job is only to be right often enough to be worth checking.
• Synthetic data generation and light labelling, with human review — high volume, individually low stakes.

Where they are not

• Multi-step agentic loops. Plan, call a tool, read the result, adapt, repeat. The compounding argument applies at full force; this is the clearest failure mode.
• Open-domain factual questions without retrieval. Less capacity means less stored knowledge and, notably, worse calibration about the boundary — small models tend to be confidently wrong rather than uncertain.
• Long-context synthesis. Nominal context windows are generous; effective use of them at this size is weaker, and information in the middle of a long input is more likely to be ignored.
• Novel reasoning. Non-routine mathematics, unusual logical structures, anything requiring a step not well represented in training.
• Subtle instruction adherence. Following six constraints at once while also doing something hard. Small models drop one, usually the least recently mentioned.
• Anything where a rare failure is expensive and nothing downstream checks the output.

Moving the boundary

Several techniques reliably shift tasks from the second list to the first, and they are cheaper than upgrading:

• Constrained decoding. Force the output to match a grammar or JSON schema at sampling time. Format failures — the most common small-model problem in production — stop being possible rather than merely becoming rarer.
• Decompose the task. Three small prompts with verification between them beat one prompt with three steps in it. A small model is far better at doing one thing than at sequencing three.
• Retrieve rather than recall. Put the facts in the context and the capacity disadvantage largely disappears.
• Few-shot examples. Small models benefit more from in-context examples than large ones do, because they are less able to infer intent from an abstract instruction.
• Fine-tune for the one task. The strongest lever: a tuned small model on a narrow task can match a general large one, and this is the main reason open small models matter commercially.
• Verify with something cheap. A schema check, a compiler, a regex, a second opinion. Small models fail more often and are much cheaper to run twice.