Does a Knowledge Cutoff Matter If You Are Using RAG?

“We use RAG, so the cutoff is irrelevant” is right about half of a system and badly wrong about the other half, and which half is which is determined by your corpus, not by the model you picked.

The claim

A training cutoff matters exactly to the extent that your retrieved passages contradict what the model already believes. Where the model has no prior, retrieval writes on a blank page and the cutoff is irrelevant. Where the model has a strong prior that the passage contradicts, the cutoff is the dominant variable in the system and no amount of retrieval quality fixes it, because retrieval already succeeded.

That reframing is the useful part. The question “how new is this model” has no answer that helps you. The question “what fraction of my corpus contradicts the public record as of some months ago” is answerable, and it tells you whether a cutoff change during a migration is a non-event or the main risk.

Where the cutoff genuinely does not matter

Take a system answering questions over internal material: an HR policy, a ticket history, a product’s own runbooks, a private contract set. Ask it what the refund window is. The model has no memorised opinion about your refund window — there is nothing in its training data on the subject, because the subject is yours. The retrieved passage is the only evidence in context, and the model uses it.

For this class of system a newer cutoff buys you nothing on the knowledge axis. It may buy you other things — better instruction following, better long-context behaviour, a cheaper price — and those are real reasons to migrate. But a team that delays a migration because the target model has an older cutoff, on a corpus the model has never seen either way, is optimising a variable that is not connected to their output.

The same holds for anything whose answer is a fact about the retrieved document rather than about the world: extraction, classification against your taxonomy, summarisation with constraints. The model is being used as a reader, and readers do not need current events.

Where it dominates

Now take a system answering questions over a public, fast-moving corpus: the documentation for a widely-used library whose API changed. You retrieve the current page, which says the method is called one thing. The model has seen tens of thousands of code samples calling the old name. It will sometimes write the old name anyway — fluently, with the correct page sitting in its context, and with no signal that anything went wrong.

This is not a retrieval failure. Recall was perfect. The failure is in generation, and the mechanism is straightforward once stated: the output is a distribution conditioned on the whole context, and a heavily reinforced prior competes with a single in-context passage rather than being overwritten by it. The passage wins more often when it is unambiguous, when it is close to the answer position, when it is one of few rather than one of twenty chunks, and when the instructions say explicitly that context overrides prior knowledge. It wins less often when it is buried, hedged, or formatted unlike anything the model expects.

Note the shape of that list: every item on it is something you control. Which is why the honest version of the claim is not “the cutoff matters” but “the cutoff sets how hard your prompt and your chunking have to work”.

The migration risk runs both ways

Here is the part that surprises teams. Migrating to a model with anewer cutoff can make a RAG system worse.

Where the old model had no prior and simply read the passage, the new model may have a confident prior about the same subject — because the subject entered its training window — and now it has something to disagree with. If your corpus has moved on since that window, or if your internal usage differs from the public convention, the newer model is the one that argues with your documents. The symptom is the one nobody diagnoses correctly: “we upgraded the model and kept the pipeline identical, and the answers got worse.”

The counterargument is obvious and correct as far as it goes: instruct the model to prefer retrieved context over its own knowledge, and require a citation for every claim. Do that. But it is not free, and pretending it is free is how the other failure arrives. A model instructed to defer absolutely will also defer to a wrong chunk, so a pipeline with mediocre retrieval precision gets worse as deference rises. Deference and precision trade against each other, which is precisely why both have to be measured rather than argued about.

The contradiction probe

The measurement is not an accuracy benchmark and should not be built like one. It measures deference: given a passage that contradicts a plausible prior, does the answer follow the passage.

1. Select documents from your own corpus that state something a general model would plausibly state differently — a renamed field, a changed threshold, an internal convention that inverts a common one, a policy that differs from the industry default. Aim for at least fifty; they are cheap to find because they are the same documents that generate support tickets.
2. For each, write one question whose answer is stated in the passage and differs from the prior, and record the passage’s answer as the reference.
3. Run each question through the real pipeline, with retrieval, at temperature 0. Score one binary per item: does the answer match the passage. Not “is the answer good” — the passage is the ground truth by construction, even where it is inconvenient.
4. Report deference as the proportion matching, per model, on the same items. Also record, separately, the proportion where the model asserted the prior and cited the retrieved document, because that combination is the one that survives human review and is the most damaging.
5. Apply the decision rule: if deference on the candidate model is lower than on the incumbent, the fix is prompt-side — an explicit override instruction and mandatory citation — not retrieval-side. Re-run the probe after the prompt change and ship only if deference is restored to at least the incumbent’s level, with end-task accuracy on your normal eval set unchanged.

Two caveats about interpreting the result. A published cutoff is an upper bound on knowledge rather than a guarantee: coverage thins towards the edge of the window, so a model can half-know a topic and produce a confident blend of the passage and a stale prior, which the binary scoring above will catch as a mismatch and which is worth reading the failures for. And deference is not a fixed property of a model; it moves with chunk count, position and instruction wording, so the number is only comparable across models when everything else in the pipeline is held constant.

The probe is worth keeping after the migration. It is the cheapest early warning that an index has gone stale relative to the world, and the same failures are the ones that make citations worth enforcing in the first place. The neighbouring question — how to compare two providers’ published cutoffs at all — is knowledge cutoff differences across a migration.