Local Model Quality: How Close to the Frontier?

“How close are open models to the frontier” has no single answer, and every article that gives one is measuring something you probably do not do. The gap is a different size on different work, and the size on your work is knowable in an afternoon.

The question as usually asked

The question assumes one quality axis. There is not one. A model that writes a clean summary may be poor at maintaining a function-calling contract across twelve turns; a model that solves competition mathematics may be worse at following a fussy formatting instruction. Aggregate scores compress all of that into a number, and the compression is lossy in precisely the direction that matters to somebody with a specific job.

It also assumes the comparison is fixed. It is not: both sides move, and open releases have historically arrived at capability levels that closed frontier models reached some months earlier — a lag rather than a plateau. Any specific figure for that lag is out of date by the time it is published, which is the reason this page gives you a method and not a number.

Where the gap is small and where it is not

This part is durable, because it follows from what the two kinds of model are optimised for rather than from a particular release.

Where open models in the mid size classes are competitive

• Bounded transformations. Summarising, rewriting, translating between formats, extracting fields from a document. There is a right answer shape and the task fits in one pass.
• Classification and routing. Especially with a few examples in the prompt. Frequently a task where a small open model is not merely adequate but preferable, because it is fast and the output space is tiny.
• Domain work after fine-tuning. A tuned open model on a narrow domain can beat a general frontier model on that domain, which is the strongest structural argument for open weights.
• High-volume, low-stakes generation where cost per call dominates and occasional imperfection is absorbed downstream.

Where the gap tends to be real

• Long multi-step agentic work. Reliability compounds: a small per-step disadvantage becomes a large per-task one over twenty tool calls.
• Hard reasoning under a long context. Holding many constraints simultaneously across tens of thousands of tokens.
• Instruction adherence under pressure. Obeying a complicated format while also doing something difficult is where frontier post-training shows.
• Breadth of world knowledge. A parameter count is a budget, and the smaller model spent less of it on obscure facts.
• Anything multimodal or tool-integrated out of the box. Often available in the open ecosystem but with more assembly required.

How to read a leaderboard

You do need a current source, because named models go stale faster than any page. Use one, and apply these corrections:

• Prefer held-out or rotating evaluations. A static public benchmark leaks into training data over time, and a score on a contaminated set measures memorisation.
• Prefer human-preference arenas for chat quality and task-specific benchmarks for task-specific quality. Do not use either for the other.
• Check the harness. The same model scores differently under different prompt templates, sampling settings and answer-extraction rules. Comparisons within one harness are meaningful; across harnesses they usually are not.
• Check what was served. A quantised deployment of an open model is not the same artefact as the reference weights. Score and serving format have to travel together.
• Read the size class, not the name. Ask what the best model under your memory budget scores, because that is the constraint you actually have.

Used this way a leaderboard produces a shortlist of three candidates, not a decision. The decision comes next.

Building the eval that decides it

Fifty examples from your own traffic beat every public benchmark for this purpose, because they are drawn from the distribution you care about and nothing has trained on them.

• Sample honestly. Take real inputs, including the ones that went wrong. An eval set of easy cases will tell you every model is fine.
• Write the grader before you look at outputs. Mechanical where possible: schema validity, required fields present, code compiles, numeric answer matches. Reserve human judgement for what cannot be checked mechanically, and write down the criterion first.
• Fix the sampling settings across all candidates — temperature 0 and an identical system prompt — so you are comparing models rather than configurations.
• Adapt the prompt per model, once. An open model given a prompt tuned for a frontier model is being tested on prompt transfer, not capability. Give each candidate one honest round of adaptation.
• Record cost and latency in the same run. A model that scores three points lower at a fifth of the cost has not lost.
• Keep the set in version control and re-run it on every model change. This is the artefact that makes the next comparison cheap, and the reason this exercise is worth doing properly once.

Making the call

Decide per task, not per company. Most systems have a distribution of request difficulty, and the useful question is what fraction of it an open model handles at your quality bar — which is usually a large fraction, with a hard tail that is not close. Route accordingly rather than picking a winner.