Why Language Models Struggle to Forecast Numbers

The standard explanation is that language models cannot do maths. That is not the interesting failure. The interesting failure is that a general chat model never receives a number as a number, and the machinery between your series and its embedding layer destroys most of what makes a number a number.

The tokenizer is the first problem

A byte-pair-encoding tokenizer is built by merging frequent character sequences in a text corpus. Digit sequences are frequent, so they get merged, and the merges follow corpus frequency rather than place value. Gruver, Qiu, Stanton and Wilson give the clean example in their 2023 paper on LLMs as zero-shot forecasters: with GPT-3’s tokenizer, 42235630 is split as [422, 35, 630]. Three tokens, and none of the boundaries is at a power of ten.

Follow what that does. The model must learn that the token 422 in first position contributes 42,200,000 while the same token elsewhere contributes 422, and that the boundary between the first and second token falls after the third digit here but somewhere else in the very next number. Adjacent values in a series are not adjacent in token space: 999 and 1000 differ by one in magnitude and by their entire token sequence, while 1999 and 2999 may share structure because both look like years. Nothing in the embedding of a digit token encodes its place value; place value has to be inferred from position, which changes per number.

The fix that works is crude and it tells you how shallow the problem is. The same authors encode each value with spaces between the digits and commas between time steps, forcing one token per digit, and drop the decimal point entirely since precision is fixed. Their worked example turns the values 0.123,1.23,12.3,123.0 into the string " 1 2 , 1 2 3 , 1 2 3 0 , 1 2 3 0 0". They also note that LLaMA tokenizes digits individually by default, which makes the added spaces unnecessary there and merely wasteful of context. A representational problem that is fixed by inserting spaces was never a reasoning problem.

Cross-entropy does not know 71 is near 72

The second problem is the objective. A language model is trained to maximise the probability of the next token under a categorical distribution over the vocabulary. A categorical distribution has no metric on its outcomes. Predicting 7 when the answer was 8 costs exactly what predicting 1 costs, because the loss sees three unordered symbols.

Every purpose-built forecaster uses a loss that does know. Squared error charges the square of the distance. Pinball loss, used for quantile forecasting, charges asymmetrically but still proportionally. A model trained on such a loss cannot make a large error cheaply; a model trained on cross-entropy can, and the only thing stopping it is that near misses are also more probable in the data, so the ordering is learned as a statistical regularity rather than enforced.

This is not a knock-down argument, and the honest version says so. Chronos, discussed in time series foundation models, also trains with categorical cross-entropy over quantisation bins and works well, so the objection is evidently survivable. But Chronos has an advantage a chat model does not: its vocabulary is 4,096 bins laid out in order along the real line, so “nearby” is at least a consistent property of the vocabulary. In a text tokenizer, the tokens that are numerically nearby are scattered arbitrarily through a vocabulary organised by text frequency.

Numbers are expensive to write down

The third problem is cost, and it is arithmetic. Take a daily series of four significant digits and suppose per-digit encoding, so four digit tokens plus one separator token per observation. One year of history is 365 × 5 = 1,825 tokens. Fifty series of one year each, in one prompt, is 91,250 tokens of pure history before any instruction. Ask for a 30-day horizon per series and the output is 50 × 30 × 5 = 7,500 tokens, generated one at a time.

Compare that with what the same information costs a purpose-built model: 365 floats per series, one position each, no separators, and a horizon emitted in a single forward pass if the head is direct multi-step. The chat model is paying roughly five tokens per number for a representation the numeric model gets for one position, and paying it on both sides of the call. Long-horizon multi-series forecasting is precisely the workload where that multiplier bites hardest.

The honest counter-argument

The strongest case against this essay is the empirical one. Gruver and colleagues report that GPT-3 and LLaMA-2, given a properly encoded series and nothing else, extrapolate at a level comparable to or exceeding purpose-built models trained on the downstream task. They attribute it to the model’s bias toward simple, repetitive continuations and to its ability to represent multimodal distributions, both of which happen to suit series with repeated seasonal structure. That result is real and it should make anyone cautious about declaring the approach unworkable.

The paper contains its own counterweight, though: it notes that GPT-4 can perform worse than GPT-3 on this task because of how it tokenizes numbers. A capability that moves backwards with a stronger model, for tokenizer reasons, is not a capability you can plan around. It is a property of a particular vendor’s vocabulary at a particular time, and vocabularies are chosen for text.

What follows from this

The position this page argues is narrow. Use a model whose loss knows what a number is when the task is numeric extrapolation, and use a language model for the parts of a forecasting problem that are actually linguistic: reading the incident log that explains last March’s outlier, mapping a free-text product description onto the attributes you pool over in cold-start forecasting, writing the note that goes with a number. Those are language tasks adjacent to a forecasting pipeline and they are not in competition with a forecaster.

The reason to hold this position firmly rather than loosely is that the alternative has a specific cost profile: it is slow, it is expensive per number, its accuracy depends on a tokenizer you do not control, and its failure mode is a plausible-looking sequence rather than an error. A seasonal naive baseline costs nothing and is available in one line. Anything proposing to replace it should be beating it on your data, measured over several origins, before it earns a dependency.