Hybrid model architectures

For about five years the answer to “what architecture is this model” was the same for everything worth calling: a decoder-only transformer with full causal attention, and the only interesting numbers were the parameter count and the context length. That has stopped being true. The models in this cluster replace some of the attention layers with state-space blocks, or restrict most layers to a fixed local window, or route each token through a fraction of the feed-forward weights — and in each case they do it for a reason you can observe from outside, in memory footprint, in how the cost of a long prompt grows, or in the gap between the size on the model card and the price on the invoice.

These pages work from the published artefact outward: what the model card or the licence actually says, and then what that choice does to you when you serve or call the thing. Architecture that has no consequence you can measure is left out. Architecture that explains why a 400-billion-parameter model is quoted at a small model’s price, or why a 256K window fits on hardware that could not hold a transformer’s KV cache at a quarter of it, is the whole subject.