The Gap Between Open and Closed, Measured Over Time

“How far behind are open models” is asked constantly and answered badly, because the usual answer is a difference between two scores. A score difference is not a gap. A gap is a duration, and measuring it as one changes what you conclude.

Why the question is hard to answer

Three things make a naive comparison untrustworthy, and all three get worse over time rather than better.

The benchmarks move. A test that discriminated well two years ago is saturated now, with every serious model above ninety per cent, and differences near the ceiling are noise. Meanwhile the newest benchmarks have no history, so any long time series is stitched together from tests that were not measuring the same thing.

The comparison is unstable in what it compares. The top closed model against the top open model tells you about two specific artefacts of wildly different size and cost. The best open model that fits on one card, against the closed model at a comparable price, is a different and usually more useful question with a different answer.

And the closed side is not a fixed target. Closed models are served behind a system that may include routing, tool use and post-processing you cannot see, and they are updated without notice under the same name. You are comparing a file you can hash against a service you cannot.

Define the gap as a lag

The formulation that survives all of this: for a given capability threshold, how long after the first closed model crossed it did the first open-weight model cross it?

for each threshold T on a fixed benchmark B:
    t_closed(T) = release date of the first closed model scoring >= T
    t_open(T)   = release date of the first open-weight model scoring >= T
    lag(T)      = t_open(T) - t_closed(T)

# the trend in lag(T) across thresholds is the answer.
# falling lag  -> the ecosystem is catching up
# flat lag     -> a stable following distance
# rising lag   -> the frontier is pulling away

A lag in months is robust to things a score difference is not. It does not care that the benchmark is saturating, because it asks when each side crossed a level rather than how far apart they are now. It is unaffected by the two sides being different sizes. And it is a quantity you can act on: a stable lag of some number of months means today’s frontier capability is a planning horizon, not a permanent advantage.

A method you can run

• Fix the harness before you collect anything. One evaluation framework, one prompt format per benchmark, one answer-extraction rule, recorded in a file. Every number must come from the same pipeline or the series is meaningless.
• Choose three benchmarks with different character — one reasoning, one code, one instruction-following or preference — and expect the lags to differ. A single number hides that code and reasoning have not tracked each other.
• Record release dates, not evaluation dates. The lag is about when capability became available, and evaluations happen whenever someone gets round to it.
• Record the licence alongside each open entry. A model that crossed a threshold under a research-only licence did not make that capability available to build on, and for many readers that is the distinction that matters.
• Record size and serving cost. A threshold crossed by a model requiring a rack is a different event from one crossed by a model that fits on one card, and the second is what most people mean.
• Add a private held-out set. Fifty of your own problems, never published. It is the only line on your chart that cannot be contaminated, and over time it becomes the most valuable one.

Four ways the chart goes wrong

• Contamination inflates the open side over time. Public benchmarks leak into training corpora, and later models score better partly by having seen the questions. Because open models are frequently trained on broad web-scraped and synthetic mixtures, this can flatter recent open entries specifically. A held-out set is the only defence.
• Saturation compresses everything at the top. Once both sides are in the nineties, the remaining points are error bars. Retire a benchmark when the leader crosses a high threshold rather than continuing to plot it.
• Cost-blindness inverts the conclusion. A chart of capability against time says the gap is real. The same data plotted as capability per unit of cost often says the open side leads, and for anyone making a build decision that is the more relevant chart. Plot both.
• Serving mismatch measures the wrong artefact. The reference weights and the four-bit quantisation people actually run are different models. If the number came from the reference weights and your deployment is quantised, your gap is larger than the chart shows.

A fifth, less a distortion than a caveat: benchmark capability is not product capability. Closed offerings bundle tool use, retrieval, multimodality and reliability engineering that a bare checkpoint does not have, and closing a benchmark gap does not close that one. Whether it matters depends entirely on whether you were going to build those parts anyway.

What to do with the answer

The useful output is not a number to quote but a planning rule. If your measured lag on the capability you need is stable and short, then anything you build today against a frontier API is likely to become runnable on weights you control within that window — which argues for keeping the model layer swappable and not much else. If the lag on your specific capability is long or widening, build for the frontier and stop paying the tax of pretending otherwise.

Either way the decision is per capability, not per ecosystem. Summarising and extraction crossed into open weights long ago; long-horizon agentic reliability has not, on any honest reading. Measure the one you need.