Hallucination Demonstrator: A Generator With No Truth Term

A real n-gram generator over text you paste, with a context slider that walks from gibberish through fluent nonsense to near-verbatim recitation.

Context length — how many previous words the generator conditions onThe one control. At 1 it is gibberish, at 2–3 it is fluent and false, at 5–6 it is reciting your text back.

2 words

Generated text — every word chosen only by what usually follows

language model does not store facts the way a database stores rows. It stores the statistics of which token tends to follow which other tokens, across a very large amount of text. When you ask it a question, it is continuing a sequence in the way a database stores rows. It stores the statistics of which token tends to follow which other tokens, across a very large amount of text.

Of the generated text, copied verbatim from your source

84%

At this context length there is almost nothing left to choose — the generator has memorised your text and is reciting it. That is the overfitting end.

Words in your source: 133
Distinct 2-word contexts: 124
Average continuations per context: 1.06
Words generated: 70
8-word windows checked: 63
…found verbatim in the source: 53
Dead ends restarted from: 0

Words to generate

Random seedFixed, so a shared link reproduces this exact passage.

Source textPaste your own — a README, a policy document, your own writing. The default is this page's own prose.

What this assumes: this is a word-level n-gram Markov chain, not a language model. It has no embeddings, no attention, no training, and roughly six orders of magnitude fewer parameters than anything you would call an LLM, so the quality of its output tells you nothing about the quality of a model’s output. What it shares with a language model is the only thing this page is claiming: the objective. Both pick the next token from a distribution over what usually follows, and neither has any term anywhere in it that refers to whether the result is true. Everything here is deterministic given the seed.

Fluent and false are not in tension

The most persistent misunderstanding about hallucination is that it is a failure — a bug, something that went wrong, something a better model will not do. It is easier to see what it actually is on a generator small enough to hold in your head. This one has a few hundred parameters. Set the context to two words and it produces sentences that are grammatical, confident and coherent for a clause or two at a time, and it does that without any component that could even represent the idea of being wrong. There is nowhere in the process for a truth check to live.

The slider also shows the other end, which is the reason the fix is not simply “memorise more”. Push the context to five or six words and the verbatim overlap climbs toward 100%: the generator stops composing and starts reciting. It is now perfectly accurate about your source text and completely useless for anything not in it. Every generative system lives somewhere on that axis, and the useful region is exactly the region where output is not traceable to a source — which is the same region where it can be wrong.

That is why grounding works and prompting does not. Telling a model to be accurate adds tokens to a distribution; it does not add a mechanism. Retrieving the relevant passage and requiring the answer to quote it adds one, because now there is something to compare against. So does asking for citations you actually resolve, and so does refusing to answer when retrieval returned nothing.

What this page cannot show you is a rate. How often a given model is wrong on your data is a measurement, and it needs your prompts, your model and a grading rubric.

Hallucination Demonstrator: A Generator With No Truth Term

Fluent and false are not in tension

The reasoning behind this

Related tools