OLMo’s Fully Open Training Data, and What “Open” Means Beyond Weights

Almost every model described as open is open in exactly one respect: you can download the weights. The Allen Institute for AI built OLMo to be open in the other respects as well, and the useful way to understand what that means is not as an adjective but as a list of files that exist for OLMo and do not exist for most of its peers.

The word open is doing two jobs

When a lab releases a model under a permissive licence, what has been released is a checkpoint: a large tensor file, a tokenizer, a config, and a card describing what the thing scores on some benchmarks. You can run it, fine-tune it, and ship it. You cannot answer any question that begins “why does it do that”, because the process that produced the tensor is not in the box.

This is not a moral complaint, it is a practical one. If a model reproduces a passage verbatim, you cannot check whether it was in the training set. If it behaves oddly on a domain, you cannot check how much of that domain it saw. If you want to make a genuinely different model rather than a fine-tune of somebody else’s, you have no recipe to vary. AI2’s framing is that a model is open when the whole pipeline is, and OLMo is the reference implementation of that claim.

What AI2 ships with an OLMo release

From AI2’s release materials and the project page at allenai.org/olmo, an OLMo release comprises, in addition to the final weights:

• The pretraining corpus. Published as a separate dataset — Dolma for OLMo 1, and the OLMo-mix and Dolmino mixes for OLMo 2 — with the documents themselves, not a description of them.
• The data-processing toolkit. The code that built the corpus: deduplication, filtering, tagging, decontamination against evaluation sets.
• The training code and configuration. The actual trainer, with the hyperparameters and the distributed setup, rather than a paragraph in a paper describing them.
• Intermediate checkpoints. Not just the final weights but hundreds of snapshots taken through training, which is what makes it possible to study when a capability appeared.
• Training logs and metrics. Loss curves and run telemetry, published rather than summarised.
• The evaluation suite. AI2 published its evaluation framework alongside the models, so the reported numbers can be regenerated rather than trusted.
• Adaptation artefacts. The instruction-tuning and preference data used to produce the instruct variants, under the Tulu line of work.

Everything in that list is released under permissive terms — the models under Apache 2.0, the data under its own terms — which is a separate axis from the licence question that dominates Falcon’s release history. A model can be Apache 2.0 and still ship nothing but a tensor.

Dolma, and why the corpus is the hard part

Dolma is the pretraining corpus AI2 built and published for OLMo, assembled from web crawl, code, academic papers, books, encyclopaedic text and social content, at a scale of trillions of tokens. It is documented in its own paper and released as a dataset with the tooling that produced it.

Publishing a corpus is much harder than publishing weights, and the reasons are worth understanding because they explain why so few labs do it. Weights are a derived artefact and their relationship to any individual training document is diffuse. A corpus is the documents. It carries the copyright status, the personal data, and the licensing questions of every source it drew from, directly and legibly. A lab that publishes its corpus is accepting scrutiny that a lab publishing only weights avoids by construction. That asymmetry, rather than any technical difficulty, is most of why “open weights” became the default meaning of open.

There is a second, quieter reason. A corpus is enormous. Dolma is measured in terabytes, and hosting it, versioning it, and keeping it downloadable for years is an ongoing cost with no revenue attached to it. Weights are a few hundred gigabytes at most and get mirrored by the community within hours. So even a lab that wanted to publish its data has an infrastructure commitment to make, and the reason AI2 can make it is that publishing research artefacts is what the institute is for. That is a structural fact about who does this rather than a gap anyone is likely to close by being persuaded.

OLMo 1, OLMo 2 and OLMoE

OLMo 1

Released in February 2024 at 1B and 7B, with a 65B model trained later. The 7B model’s documented context length is 2,048 tokens, which is short by any current standard and reflects both the era and the project’s priorities: the point was a fully reproducible pipeline, not a long-context record.

OLMo 2

Released from late 2024 at 7B and 13B, with a 32B model following. The documented context length moved to 4,096 tokens, and the release added the mid-training stage and the Tulu-derived post-training recipe. AI2 positioned the 32B model as the first fully open model — open in the full sense described above — to be competitive with contemporary closed models of similar scale on their published evaluation suite.

OLMoE

A mixture-of-experts variant, roughly 7B total parameters with about 1B active per token, released with the same full set of artefacts. It is the only sparse model whose complete training pipeline is public, which makes it the one place you can actually inspect how expert routing develops during training rather than inferring it from a served endpoint. If you want the inference-side consequences of that architecture, they are the same ones described for DBRX and Qwen 3.

When the difference actually affects you

Most production work does not need the corpus. If you are calling a model through an API to summarise support tickets, the training data is not a variable you can act on. There are four situations where it becomes one:

• Contamination questions. If a model scores suspiciously well on your evaluation set, the only definitive check is searching the training corpus for it. With OLMo you can. Everywhere else you are reasoning from decontamination claims you cannot verify.
• Provenance and audit. Regulatory and procurement regimes increasingly ask what a model was trained on. “The vendor says it was lawfully sourced” is a different answer from a dataset you can point at.
• Research on training dynamics. Anything asking when or why a behaviour emerged needs intermediate checkpoints, and those are simply absent from an open-weight release.
• Building a model rather than tuning one. A published recipe with published data is a starting point you can vary. A checkpoint is not.

There is a fifth case that is easy to miss, and it is the one most likely to apply to a working team: OLMo is the only model where you can answer a question about behaviour by looking rather than by guessing. When a model does something odd on a domain — a consistent formatting habit, a strange refusal, an unexpected competence — every explanation you can offer for a closed-weight model is a hypothesis. With the corpus and the checkpoints in hand it is a search. That will not usually be worth doing on a production model you did not train, but it makes OLMo the right thing to reach for when you are trying to understand a class of behaviour rather than ship a feature.

If none of those describes you, an open-weight model is fine and the distinction is academic. If one of them does, it is the whole question, and OLMo is currently the clearest example of the alternative existing at all.