Next-Token Prediction: A Model Small Enough to Check
Counts a real n-gram language model from your own text in the browser and shows the top ten candidates with the raw counts behind every probability.
40.0% of 5 observed continuations, over 4 distinct tokens.
- Tokens counted in your corpus
- 138
- Distinct tokens (the vocabulary)
- 57
- Context the model used
- 2 tokens
- Times that context occurs
- 5
- Distinct continuations seen
- 4
- Entropy over the continuations
- 1.922 bits
- Contexts at this length with exactly one continuation
- 90.3%
- Longest run in the sample below that is verbatim from your corpus
- 16 tokens
a ticket is assigned an escalated ticket answers it the same day . the person who answers a ticket is reassigned . a manager who is assigned
What this shows and what it does not
Autoregressive generation is one operation repeated: given everything so far, put a probability on every token that could come next, pick one, append it, repeat. That is the whole loop, and it is the loop running above. What separates this from a frontier model is not the loop and not the output format — it is how the distribution is obtained. Here it is a lookup in a table of counts. There it is a forward pass through a hundred billion parameters that generalises to contexts never seen in training.
Drag the context length and watch the trade that defines the whole field. At zero context the model knows only which words are common; the output is grammatical noise. At two or three tokens it starts producing phrases that sound like your corpus. Keep going and watch the last two rows of the breakdown: the share of contexts with exactly one possible continuation climbs towards 100%, and the longest verbatim run climbs with it. The model stops predicting and starts reciting. It has not become better; it has run out of data at that context length and is reproducing its training set. Every generalisation claim about a large model is a claim that it landed somewhere between these two failure modes.
The honest limitation is the zero. When a context has never occurred, this model has nothing at all to say, and it tells you so. A neural language model never says that: it always returns a full distribution over every token in its vocabulary, however unfamiliar the context, because a softmax over a real-valued vector cannot return "no idea". That is a large part of why models produce fluent wrong answers rather than silence, and you can see the mechanism most clearly in the one place it is absent.