Why LLMs Hallucinate: The Mechanical Explanation

A model that hallucinates is not malfunctioning. It is doing exactly the thing it was optimised to do, on an input where that thing and the truth come apart. Once you can see where they come apart, the behaviour stops being mysterious and starts being something you can engineer around.

There is no truth channel

A language model maps a token sequence to a vector of scores over its vocabulary. Softmax turns those into a probability distribution, a sampler picks one token, and the loop runs again. Every number in that vector answers one question: how likely is this token to come next, given everything before it and given the distribution the model was fitted to.

Nowhere in that computation is there a second number saying whether the resulting sentence is true. There is no field for it, no head that predicts it, and no place in the loss where it could have been supervised. Truth is a property of a claim’s relationship to the world; the model has no access to the world, only to a corpus and to your context window. What it has is a very good model of what text follows what text — and true text and plausible-sounding false text look almost identical at the token level, because the false text was assembled out of the same fragments.

This is why fluency is not evidence. A confidently phrased fabrication is a high-probability continuation that happens to be false. It is not the model straining; it is the model succeeding at its actual objective.

What the pretraining objective rewards

Pretraining minimises cross-entropy: for each position, raise the probability the model assigned to the token that actually appeared. Averaged over a corpus, this is density estimation. The model is being asked to become a good approximation of the distribution of text on the internet, in books, in code.

Two consequences fall out immediately. First, the corpus itself contains false statements, and a faithful density model reproduces them — Lin, Hilton and Evans built TruthfulQA (2022) around exactly this class, which they call imitative falsehoods, and reported the uncomfortable result that on their benchmark larger models were in some cases less truthful, because they imitate the distribution better.

Second, and less obvious: the corpus is overwhelmingly written in the register of someone who knows. Encyclopaedia entries, documentation, answers on forums, textbooks. Text that says “I don’t know” is rare, and text that says “I don’t know” in the specific place where this particular model happens to be uncertain is rarer still, because the corpus was not written with reference to any model’s ignorance. So the highest-probability continuation of a question is nearly always an answer, not an admission.

Facts seen once: a lower bound

Kalai and Vempala’s Calibrated Language Models Must Hallucinate (2024) makes this sharper than intuition can. Their argument runs through the Good–Turing missing-mass estimator, the classical statistic for “how much probability belongs to things you have not seen enough of”. Consider the arbitrary facts in a corpus — a person’s birthday, a paper’s year, a function’s exact signature — and in particular the ones that appear exactly once. Those are the monofacts.

The result is that a model which is well calibrated as a generative model of the corpus must produce false statements about that class of facts at a rate related to the monofact fraction. Not because it was trained badly, but because calibration and abstention pull in opposite directions: a model that refuses to emit anything it saw only once would be systematically under-generating a slice of the distribution it was fitted to. Hallucination on rare facts is, in that framing, the price of being a good density model.

The practical reading: the failure concentrates where the training signal was thinnest. Obscure people, small libraries, recent events, internal identifiers, anything long-tail. That is a prediction you can check against your own logs, and it usually holds.

Why guessing beats abstaining

Pretraining explains where errors come from. It does not explain why post-trained assistants remain so willing to state them. Kalai, Nachum, Vempala and Zhang’s Why Language Models Hallucinate (2025) supplies the second half, and it is an argument about scoring rather than about architecture.

Almost every benchmark a model is tuned against grades a response binary: correct or not. Under binary grading, abstention scores zero with certainty. A guess scores zero most of the time and one occasionally. So for any positive probability of being right, guessing has strictly higher expected score than saying “I don’t know”. Optimising against those scoreboards — through reinforcement learning from human feedback, through leaderboard-driven model selection, through the everyday preference of raters for an answer over a hedge — therefore trains the hedge out.

This is the part most explanations miss, and it matters because it means hallucination is partly a choice of metric, not only a property of the network. The paper’s own suggestion is to change the scoring: state a confidence threshold in the task itself and penalise errors relative to it, so that abstention becomes the rational move below the threshold. That idea is the whole subject of making a model abstain instead of guessing.

What follows for your system

• You cannot prompt it away. “Do not make things up” changes the conditional distribution slightly. It does not add a truth channel. It shifts style more reliably than it shifts accuracy.
• Grounding is the structural fix. If the claim must be traceable to a span in the context, you have replaced a question about the world with a question about a document — and that one the model can answer, and a verifier can check.
• Uncertainty signal exists, weakly. The token distribution is not a truth channel, but it is not noise either. Logprobs, sampled-answer disagreement and self-evaluation all carry usable signal, which is the subject of the calibration and detection pages in this cluster.
• The rate is a property of your prompts. It moves with domain, retrieval quality, prompt shape and model. Any single published percentage is about somebody else’s workload.