Temporal Fusion Transformer Explained

The Temporal Fusion Transformer is often described as a transformer applied to time series. It is closer to an LSTM sequence-to-sequence model with three learned selection mechanisms wrapped around it, only one of which is attention. Knowing which is which tells you what it can and cannot explain about your data.

Three kinds of input, and why that matters

Bryan Lim, Sercan Arık, Nicolas Loeff and Tomas Pfister published the architecture as “Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting” in 2019. Its most useful contribution is arguably not a layer at all but the way it partitions inputs, because that partition is what makes it fit real planning data.

• Static covariates — attributes that do not change with time: store region, product category, shelf position. Available at every step, past and future.
• Known-future inputs — time-varying values you know in advance: the calendar, a scheduled promotion, a price you have already set, a public holiday. These are usable at forecast time across the whole horizon.
• Observed past-only inputs — time-varying values you only learn as they happen: the target itself, actual traffic, actual weather. These stop at the forecast origin.

Most architectures collapse the last two into one bucket, which forces you to either forecast your covariates or discard them. Keeping them separate is what lets the decoder legitimately use next month’s promotion calendar. It is also the source of the most common bug in a TFT pipeline: putting a variable in the known-future bucket that you will not actually know at inference time. Training accuracy looks excellent, production accuracy collapses, and nothing errors.

Variable selection networks

Each of the three input groups gets its own variable selection network. At every time step it produces softmax weights over the available variables and takes a weighted combination of their transformed representations. Each variable has its own gated residual network for the transformation, with weights shared across time steps.

Two things this buys. Irrelevant inputs get near-zero weight, so throwing forty candidate covariates at the model is not the disaster it would be in a linear regression. And the weights are readable — they are per-instance, so you can ask which variables mattered for this store in this week rather than only in aggregate.

The care needed here is the same care needed with any attribution. Softmax weights are competitive: two correlated covariates split the weight between them arbitrarily, and dropping one raises the other’s weight without anything having changed causally. Read them as “the model used this”, never as “this drives demand”.

Gated residual networks

The gated residual network is the block used everywhere in the architecture, and its job is to make depth optional. It applies a non-linear transformation, passes the result through a gated linear unit, and adds a residual connection — so the gate can drive the non-linear branch to zero and leave the input passing through untouched. The paper’s own framing is that this lets the model skip over any unused components.

That matters more in forecasting than in most domains because dataset sizes vary so much. The same architecture is asked to handle a few hundred series and a few hundred thousand, and on the small end the capacity is a liability. A gate that can switch off a non-linearity means the model degrades gracefully towards something nearly linear instead of overfitting — which is the same conclusion the observations-to-parameters argument arrives at from the other direction.

Static covariates feed the rest of the network through four separate context vectors produced by static covariate encoders: one conditions the temporal variable selection, two initialise the LSTM’s cell and hidden state, and one enriches the temporal features after the sequence layer. The point of routing static information through four paths rather than concatenating it once is that “this is a small rural store” should change which variables matter, where the sequence starts from, and how the temporal features are read — different mechanisms, so different injection points.

Between the selection layers and the attention sits an LSTM encoder-decoder: the encoder consumes the past inputs, the decoder runs over the horizon. It handles local ordering, which is why the architecture needs no positional encoding — the recurrence supplies the sequence ordering that a plain transformer has to be told about.

Interpretable multi-head attention

This is the only attention in the model, and it is modified from the standard version specifically so its weights can be read. Standard multi-head attention gives each head its own value projection and concatenates the heads, so there is no single attention distribution to look at — head three’s weights and head five’s weights are over different value spaces and averaging them is meaningless.

The TFT shares the value projection across all heads and aggregates the heads additively rather than by concatenation. Because every head now weights the same values, the per-head weights sum into one distribution over time steps that means something: how much this forecast drew on each past step. That is what makes the published interpretability use cases — identifying persistent temporal patterns, and spotting regime changes as shifts in attention — possible at all.

The trade is capacity. Independent value projections let different heads attend to genuinely different content; sharing them means the heads can only differ in where they look, not in what they extract. The authors accepted that cost to get a readable attention map, which is a defensible choice and worth knowing you have made.

Quantile outputs, and where it breaks

The model is trained on a quantile loss and emits several quantiles per horizon step — the experiments use the 10th, 50th and 90th percentiles. So the native output is an interval, not a point, which is the right shape for an inventory or staffing decision. The mechanics of that loss are on quantile regression for forecasting, including the trap that separately-estimated quantiles can cross.

Where the architecture disappoints, it tends to be for one of these reasons:

• Not enough series. Every criticism of large global models applies, and the TFT is large. Below a few thousand series with a few hundred observations each, a gradient-boosted tree with calendar features is the stronger baseline and trains in minutes.
• Leaked known-future inputs. Covered above, and worth repeating because it is silent. Audit the known-future list against what your pipeline can actually supply at the forecast origin.
• Interval calibration is empirical. The quantiles are as calibrated as the training distribution. After a regime change they are wrong in a way the model cannot detect — see detecting structural breaks for the diagnosis.
• Attention over long lookbacks is quadratic. The attention cost grows with the square of the sequence length, so a year of hourly history is an expensive lookback. Coarsening the input granularity is usually the cheaper fix than a longer window.