Continued Pre-Training for Domain Adaptation

Continued pre-training is not a bigger fine-tune. It is the same objective pretraining used — next-token prediction on raw text — run for a further large number of tokens on a corpus you chose. The difference from SFT is three orders of magnitude of compute, and that gap is worth deriving before anyone commits to it.

When it is the right instrument

Three conditions have to hold together. Any one alone points somewhere cheaper.

• The corpus is large and unlabelled. Hundreds of millions to hundreds of billions of tokens of domain text with no instruction-response structure. You cannot use it for SFT because there are no targets.
• The distribution is genuinely far from pretraining. A language poorly represented in the base corpus, a specialised notation, protein sequences, a proprietary codebase in an unusual style. “Legal documents” usually does not qualify — English legal text is well represented in web-scale corpora.
• The tokeniser is a problem. If your domain text fragments into far more tokens than English does, you are paying that penalty on every request forever, and it is a signal the base model has poor coverage of your distribution.

The tokeniser test is concrete and takes ten minutes: tokenise a thousand representative documents with the base tokeniser and compute tokens per character. Compare with English. A ratio far above it means the model is spending capacity on segmentation rather than meaning.

Before committing, rule out the cheaper alternative properly. Most domain corpora can be converted into instruction data — take a document, generate questions it answers, and you have an SFT set from material you were going to feed to a pretraining run. That path is three orders of magnitude cheaper by the arithmetic below, and for domains that are merely specialised rather than genuinely out-of-distribution it usually wins. Continued pre-training earns its budget when the text itself is unlike anything in the base corpus, not when it is ordinary prose about an unusual subject.

The compute budget, derived

The standard approximation for transformer training compute is C ≈ 6·N·D FLOPs, where N is parameters and D is training tokens — one unit for the forward pass and two for the backward, times two FLOPs per multiply-accumulate. It is an approximation and it is close enough to plan with.

CONTINUED PRE-TRAINING
  N = 7e9 parameters,  D = 5e11 tokens (500B)
  C = 6 · 7e9 · 5e11              = 2.10e22 FLOPs

  on an accelerator with 3.12e14 FLOP/s dense bf16 peak,
  at 40% model FLOP utilisation   = 1.25e14 FLOP/s effective

  time = 2.10e22 / 1.25e14        = 1.68e8 seconds
                                  = 46,700 GPU-hours
                                  ≈ 5.3 GPU-years
                                  ≈ 12 days on 160 GPUs

SUPERVISED FINE-TUNING, same model
  50,000 examples x 1,000 tokens x 3 epochs
  D = 1.5e8 tokens
  C = 6 · 7e9 · 1.5e8             = 6.30e18 FLOPs
  time = 6.30e18 / 1.25e14        = 50,400 seconds
                                  ≈ 14 GPU-hours

RATIO                              ≈ 3,300x

Two caveats on those numbers. The 40% utilisation figure is a reasonable planning assumption for a well-tuned distributed run and will be lower for a first attempt. And the SFT line assumes full fine-tuning; a LoRA run has a cheaper backward pass and lands lower still. Neither caveat changes the conclusion, which is that these are not the same kind of project and should not be in the same budget line.

The recipe the literature converged on

The main practical questions — how to set the learning rate when resuming from a converged checkpoint, and how to avoid destroying general ability — have received direct attention. Gupta et al. (2023, arXiv 2308.04014) studied re-warming and re-decaying the learning rate for continual pre-training, and Ibrahim et al. (2024, arXiv 2403.08763, Simple and Scalable Strategies to Continually Pre-train Large Language Models) combined learning-rate re-warming with replay of the original distribution. The recipe that comes out of that work:

• Re-warm the learning rate, then decay it again. The base checkpoint sits at the bottom of a cosine schedule with an effectively tiny learning rate. Resuming there means almost nothing happens. Warm up to a fraction of the original peak and run a fresh decay.
• Replay the original distribution. Mix a single-digit-percentage share of general web text into the domain corpus. This is the same idea as PPO-ptx in the RLHF pipeline and the same idea as the general-data mix in forgetting mitigation. It is the most reliably useful trick in this area.
• Do not skip the instruction tuning afterwards. Continued pre-training on raw text degrades instruction following, because raw text contains no instructions. The output of this stage is a better base model, not a better assistant, and it needs SFT on top.
• Checkpoint often and evaluate general ability at each one. This is a long run and the point at which domain gains stop outrunning general losses is something you find by looking.

Extending the tokeniser

If the tokeniser test failed, adding domain tokens is part of the job and it has a specific hazard. New tokens mean new rows in the input embedding and output projection matrices, and those rows are randomly initialised — the model has no idea what they mean.

The consequences are mechanical. Randomly initialised embedding rows produce large gradients that can destabilise the early steps; the standard mitigation is to initialise each new token’s embedding as the mean of the embeddings of the sub-tokens it replaces, which starts it somewhere semantically sensible. Any adapter or evaluation artefact from before the extension is now invalid, because the vocabulary changed. And every downstream consumer — serving stack, adapters, tokeniser files — must be updated together, since a mismatch produces silent garbage rather than an error.

Published examples

• Code Llama (Rozière et al., 2023, arXiv 2308.12950). Continued pre-training of Llama 2 on a code-heavy corpus — the paper describes training on hundreds of billions of additional tokens — followed by long-context fine-tuning and instruction tuning. The clearest public example of the full pipeline, including the fact that the general instruction stage came after.
• Domain-adapted model families in medicine, law and finance have followed the same pattern: take a strong open base, continue pre-training on a curated domain corpus, then instruction tune. Read the specific model card for corpus size and token counts rather than trusting a summary — they vary enormously, and some projects labelled “continued pretraining” used token budgets closer to a large fine-tune.

The decision rule that follows from the derivation: if you are considering continued pre-training and your corpus is under a billion tokens, run the SFT experiments first. At that size you are in the region where a well-curated instruction dataset is both cheaper and more likely to move the metric you care about.