When a Classical Model Beats a Transformer at Forecasting

The honest answer to “should I use a transformer for forecasting” almost never depends on the architecture. It depends on whether you have enough series to learn shared structure from, and that is a number you can compute before you write any modelling code.

The question is not which is better

Exponential smoothing and ARIMA are local models: one fit per series, a handful of parameters, and no information shared between series. A deep forecaster is a global model: one set of weights fit across every series at once, so it learns what a promotion spike looks like from all the SKUs that ever had one and applies that to a SKU that has never had one.

That is the actual difference, and it is not a difference in sophistication. A global model has one enormous advantage — cross-series learning — and one enormous requirement, which is enough series to learn from. Where the requirement is not met, the advantage is not available, and what is left is a model with a hundred thousand parameters trying to fit a series with eighty observations. It is not that classical “beats” the transformer there. It is that the transformer has been given nothing to be good at.

The ratio that decides it

A global model’s training signal is the total number of forecast targets across all series: series count times observations per series, minus the lookback window you burn at the start of each. Compare that to the parameter count of the model you are proposing.

targets  =  N_series  ×  ( T_observations  −  lookback  −  horizon + 1 )
ratio    =  targets / parameters

Work it for a concrete case. A regional retailer with 200 SKUs and three years of weekly data has T = 156. With a lookback of 52 weeks and a horizon of 13, each series contributes 156 − 52 − 13 + 1 = 92 targets, so:

targets    = 200 × 92          = 18,400
parameters ≈ 100,000 – 1,000,000   (a small TFT or N-BEATS configuration)
ratio      = 0.02 – 0.18 targets per parameter

Fewer targets than parameters, by one to two orders of magnitude. That is a model that can memorise its training set exactly, and no amount of regularisation turns a ratio like that into learned structure. Now the same arithmetic for a chain with 40,000 SKUs and three years of daily data (T = 1,095, lookback 90, horizon 28):

per-series targets = 1,095 − 90 − 28 + 1 = 978
targets            = 40,000 × 978 = 39,120,000
ratio              = 39 – 391 targets per parameter

Three to four orders of magnitude apart from the first case, from the same formula. The threshold is not a specific number — it depends on how correlated your series are, since a thousand near-identical SKUs carry much less information than a thousand different ones — but the gap between those two situations is not a matter of judgement. If your ratio is below one, the argument is over.

The ratio also tells you what to do when it is marginal: shrink the model rather than the ambition. N-BEATS and the Temporal Fusion Transformer both scale down, and a global model with ten thousand parameters trained across 200 series can still beat 200 independent ARIMA fits, because the cross-learning benefit does not require the model to be large. It is the deep configurations that require the data.

What the competitions actually showed

The Makridakis competitions are the closest thing this field has to a controlled comparison, because everyone forecasts the same series with the same scoring rule and the submissions are frozen before the outcomes are known.

• M4 (2018), 100,000 series. Pure machine-learning submissions performed poorly. The winner was Slawek Smyl’s hybrid, which put exponential smoothing inside a recurrent network rather than replacing it. Oreshkin and colleagues later reported that N-BEATS improved on that winner by 3% and on the statistical benchmark by 11% — their paper states both figures. Note the size of the win: 3% over a hybrid, on a hundred thousand series. That is a real result and it is not a rout.
• M5 (2020), 42,840 Walmart series. Gradient-boosted trees dominated, not transformers and not exponential smoothing. The data was hierarchical retail sales — 3,049 items across 10 stores — and the winning approach was feature engineering plus LightGBM. This is the case most people actually have, and the answer was neither end of the debate.
• DLinear (AAAI 2023). Zeng, Chen, Zhang and Xu showed that a single linear layer over a decomposed input matched or beat a series of published transformer forecasters on the standard long-horizon benchmarks. The paper is “Are Transformers Effective for Time Series Forecasting?” and its finding is narrower than the title implies — it is about a particular family of benchmarks and a particular set of architectures — but the narrow finding is still that a linear model was not beaten.

Read together, these do not say deep forecasting does not work. They say the margins are small, the baselines are strong, and the field has repeatedly published transformer results that a simpler model matches. Which is a reason to make the simple model your baseline and require the complex one to beat it, rather than the other way round.

Where classical genuinely wins

• Few series, or series that share nothing. The ratio above. Twelve regional demand curves with different drivers are twelve problems, not one problem with twelve instances.
• Short history. An ETS fit needs roughly two seasonal cycles to identify a seasonal component. A global neural model needs that in every series it hopes to generalise across, and it needs many of them.
• Intervals with a known basis. A state-space ETS gives you a forecast distribution derived from its own error structure. A neural quantile forecast gives you empirical quantiles that are only as good as the calibration of the training set — fine, but a different kind of guarantee, and it degrades silently when the regime shifts.
• Anything that has to be explained to a planner. A decomposition into level, trend and seasonal terms is auditable line by line. Attention weights are attributions, not explanations, and a demand planner who overrides a forecast needs to know which of those three moved.
• Sparse and intermittent series. A mostly-zero series has almost no signal per observation, and squared-error training drives a neural model straight to the mean. The specialised methods on intermittent demand exist because this is a different problem, not a harder one.

Where the deep model earns its cost

Three situations, all of which are about structure a local model cannot represent rather than about capacity.

Cold start. A new SKU has no history, so a local model has nothing to fit. A global model can forecast it from its static attributes on day one because it learned what that category looks like from every other SKU in it. This is not a marginal accuracy win, it is the difference between a forecast and no forecast — see cold-start forecasting.

Many covariates with non-linear interactions. Price, promotion, weather, holiday proximity and competitor activity interacting is exactly what regression with a handful of terms handles badly and a learned model handles well. Notably, gradient-boosted trees do this too, and won M5 doing it.

Shared seasonality across a large catalogue. Estimating a weekly pattern independently per SKU on sparse data gives you 40,000 noisy estimates. Estimating it once across the catalogue, with per-SKU deviations, is a smaller and better-posed problem, and it is what a global model does automatically.

The order of operations that follows: fit a seasonal naive baseline, then ETS or ARIMA, then gradient boosting with covariates, then a deep global model — and stop at the first one whose successor cannot beat it on a rolling-origin evaluation by a margin that survives the variance of the estimate. Most catalogues stop at step three.