Time Series Forecasting: Baselines Before Models

A forecast is not good because its error is small. It is good because its error is smaller than the error of repeating last week. That comparison has a standard name and a standard number — MASE — and this page computes it by hand on fourteen days of data before writing a line of modelling code.

The bar, and why it is not zero

Every forecasting project starts with someone reporting a mean absolute error. On its own that number means nothing: an MAE of 7 units is excellent on a series that swings by 200 and useless on one that swings by 3. The only interpretable form is a ratio against a forecast that required no work.

The reason this matters more in forecasting than elsewhere is that the trivial forecast is unusually strong. Yesterday’s value is a very good prediction of today’s. Last Tuesday’s is a very good prediction of this Tuesday’s. A model has to add something on top of an already-accurate reference, and a great many do not.

Four baselines, in order of difficulty

Choose the baseline by the structure of the series, not by which one the model beats. A daily series with a weekly cycle gets the seasonal naive; using the plain naive there is choosing an easy opponent, and it is the most common way a forecasting result is quietly inflated.

Two weeks, worked

Four weeks of daily units. Weeks 1–3 are training, week 4 is the test window. Invented numbers, chosen so the arithmetic is checkable.

         Mon  Tue  Wed  Thu  Fri  Sat  Sun
week 1    95  100   98  112  150  200   82
week 2   102  106  101  116  155  205   88
week 3   100  110  105  120  160  210   90
week 4   108  104  111  126  152  224   85   <- test

First the denominator. MASE divides the forecast’s test MAE by the in-sample MAE of the seasonal naive forecast — how well “same day last week” did on the training data. Over weeks 2 and 3 that is fourteen one-step errors:

week 2 vs week 1:   7   6   3   4   5   5   6     abs sum = 36
week 3 vs week 2:  -2   4   4   4   5   5   2     abs sum = 26

in-sample seasonal-naive MAE = (36 + 26) / 14 = 62 / 14 = 4.4286

Now the two forecasts for week 4. The seasonal naive forecast is literally week 3. The model forecast is whatever your model produced; the numbers below stand in for it.

day    actual   seasonal naive   |err|      model   |err|
Mon      108       100            8         112       4
Tue      104       110            6         108       4
Wed      111       105            6         110       1
Thu      126       120            6         122       4
Fri      152       160            8         158       6
Sat      224       210           14         205      19
Sun       85        90            5          95      10
                          sum =  53                  48

seasonal naive test MAE = 53 / 7 = 7.5714
model test MAE          = 48 / 7 = 6.8571

MASE(seasonal naive) = 7.5714 / 4.4286 = 1.710
MASE(model)          = 6.8571 / 4.4286 = 1.548

relative MAE (model / baseline, same window) = 6.8571 / 7.5714 = 0.906

Two readings, both worth having. The model is 9.4% better than the baseline on this window. And both are worse on the test window than the seasonal naive was in-sample (MASE above 1), which says the test week was harder than the training weeks rather than that the model is broken — a distinction the relative number alone cannot make.

What the aggregate number hid

Look again at Saturday. The baseline was off by 14 units; the model was off by 19. Friday and Saturday together carry 376 of the week’s 910 units — 41% of the volume — and on those two days the baseline totals 22 units of error against the model’s 25.

So the model wins the week by being better on the five quiet days and worse on the two that matter. Whether that is a good trade depends entirely on what an error costs, which is a business question and not a metric question. If a stockout on Saturday costs four times what an overstock on Tuesday costs, this model is worse than doing nothing — and the asymmetric-loss page is where that gets fixed.

The general rule: never report a single aggregate error without also reporting it split by the dimension that carries the money. Day of week, store, SKU class, high-volume decile. An aggregate improvement that reverses on the top decile is common and is almost never noticed.

Backtesting: rolling origin, not a random split

A random train/test split on a time series is not a weak evaluation. It is an invalid one: it trains on Thursday and tests on Wednesday, and the score it produces is unrelated to anything you will observe in production. This is the temporal leakage pattern and it is the single most common error in the field.

The correct evaluation is a rolling origin: fit on everything up to time t, forecast the next h periods, score, advance t, repeat. Every fit sees only its own past, and you get a distribution of errors rather than one number.

import numpy as np
import pandas as pd

def seasonal_naive(train: pd.Series, h: int, season: int) -> np.ndarray:
    """Forecast h steps by repeating the last full season of train."""
    last = train.iloc[-season:].to_numpy()
    return np.resize(last, h)

def rolling_origin(y: pd.Series, h: int, season: int, min_train: int,
                   fit_predict) -> pd.DataFrame:
    rows = []
    for t in range(min_train, len(y) - h + 1):
        train, actual = y.iloc[:t], y.iloc[t:t + h].to_numpy()
        base = seasonal_naive(train, h, season)
        pred = fit_predict(train, h)
        rows.append({
            "origin": y.index[t],
            "mae_model": np.abs(actual - pred).mean(),
            "mae_base": np.abs(actual - base).mean(),
        })
    out = pd.DataFrame(rows)
    # MASE denominator: in-sample seasonal-naive MAE over the whole history
    denom = np.abs(y.to_numpy()[season:] - y.to_numpy()[:-season]).mean()
    out["mase_model"] = out["mae_model"] / denom
    out["mase_base"] = out["mae_base"] / denom
    return out

# usage: fit_predict(train, h) returns an array of h forecasts
res = rolling_origin(y, h=7, season=7, min_train=56, fit_predict=my_model)
print(res[["mase_base", "mase_model"]].describe())

Report the median and the worst quartile of mase_model, not the mean. A forecast that is usually fine and occasionally catastrophic is exactly the forecast an average hides, and it is the one that empties a warehouse.

When you have earned the right to a model

Once the baseline is in place and backtested, the ladder is roughly:

1. Exponential smoothing / ETS. Handles level, trend and seasonality with a handful of parameters, fits in milliseconds, and is genuinely competitive on short univariate series. In statsmodels this is ExponentialSmoothing.
2. Regression on calendar features. Turn the timestamp into day-of-week, week-of-year, holiday flags, promotion flags and lagged values, then fit a gradient-boosted model on the table. This is where forecasting rejoins the rest of this cluster, and where external drivers — price, weather, marketing spend — can enter at all.
3. Global models across many series. One model fitted over all SKUs or all stores at once, with the series identity as a feature. This is usually the step that beats per-series models, because a thousand short series jointly contain more information than any of them does alone.
4. Deep sequence models. Worth trying when you have hundreds of long, related series and a team to maintain them. Not worth trying on one series of two years of daily data, where the ETS model will win and take four seconds to fit.

At every rung, the score to report is the MASE from the rolling-origin backtest against the same seasonal-naive denominator. Changing the denominator between rungs makes the ladder meaningless.

What this page does not know

It is widely repeated that most production forecasts fail to beat the seasonal naive baseline. That claim is plausible — the M-competitions run by Spyros Makridakis have made the general point about simple methods for decades — but there is no census of production forecasting systems, nobody has audited a representative sample of them, and this page is not going to invent a percentage.

What can be said without inventing anything: the comparison is cheap, it takes about twenty lines, and a team that has not run it does not know which side of the bar it is on. Run it on your own series and the question stops being a matter of opinion. If your MASE is above 1, the honest report is that the model is not yet earning its maintenance, and the useful next step is deciding whether it should exist rather than tuning it.