Hybrid Architectures: Attention Plus Something Else

Almost every architecture announced since 2024 mixes layer types rather than committing to one. The reason is a single calculation: at long context the KV cache costs more memory than the model weights, and replacing most of the attention layers with constant-state ones removes almost all of it.

The arithmetic that forces it

KV cache size follows directly from the shapes, with no benchmark involved:

bytes_per_token = 2 (K and V) * layers * kv_heads * head_dim * bytes

A 7B-class model: 32 layers, 8 KV heads (grouped-query),
head_dim 128, fp16:

  2 * 32 * 8 * 128 * 2 = 131,072 bytes = 128 KB per token

    8,192 tokens ->  1.0 GB
   32,768 tokens ->  4.0 GB
  131,072 tokens -> 16.0 GB

Model weights at fp16: about 14 GB.

At 128k context the cache for ONE request exceeds the weights.

Now replace seven attention layers in every eight with a constant-state layer, leaving four full-attention layers out of thirty-two:

  2 * 4 * 8 * 128 * 2 = 16,384 bytes = 16 KB per token

  131,072 tokens -> 2.0 GB       (was 16.0 GB)

plus the recurrent state of the 28 other layers: a few MB, flat.

An 8x reduction, and a machine that held 4 concurrent 128k
sessions now holds about 33.

That is the whole motivation, and it is arithmetic rather than a finding. Everything else on this page is about what you give up to collect it.

The layer types being mixed

The first three are alternatives for the same slot; the fourth stacks with any of them. A current design is a choice of sequence-mixing layer per position in the stack, plus a choice of feed-forward block.

Why the attention count never goes to zero

The recurring finding across the published hybrids is that a small number of full-attention layers is enough to preserve long-range exact recall, and that zero is not.

The mechanism is straightforward once stated. A constant-state layer has to decide what to keep at the moment it sees a token, without knowing what will be asked later. A full-attention layer defers that decision: it keeps everything and decides at query time. For anything resembling “find the identifier mentioned forty thousand tokens ago” — retrieval from a long document, following a reference in code, quoting a clause — the deferred decision is the one that works, and no amount of state size substitutes for it, because the problem is the ordering, not the capacity.

A handful of such layers appears to be sufficient, because once a few layers can bring a distant token’s content into the residual stream at the current position, the recurrent layers above can work with it locally.

The other levers on the same number

Mixing layer types attacks one term in the formula. Every other term has been attacked too, and because they multiply, a current model can advertise a context length that would have been unservable two years ago on the same hardware.

They compound, and the compounding is worth seeing once:

Start: 32 layers, 32 full attention heads, head_dim 128, fp16
  2 * 32 * 32 * 128 * 2       = 524,288 bytes/token  (512 KB)

Grouped-query, 8 KV heads     = 131,072  (128 KB)   4x
Hybrid, 4 attention layers    =  16,384  ( 16 KB)   8x
8-bit KV cache                =   8,192  (  8 KB)   2x
                                --------
                                total reduction: 64x

At 128k tokens:  64 GB  ->  1 GB.

None of those four is a research risk any more, and none of them requires the others. That is the actual reason long context stopped being exotic, and it is a better explanation than any single architectural claim.

What shipped models actually do

• Sliding window throughout. Mistral 7B (2023) used a 4,096-token sliding window in every layer, with the argument that information propagates beyond the window across layers — at 32 layers, a theoretical reach of 32 windows.
• Alternating local and global. Gemma 2 (Google, 2024) alternates sliding-window layers with full-attention layers, roughly one in two, which caps most of the cache while keeping frequent exact access.
• Interleaved local and global with cross-layer sharing. Character.AI published a serving-focused design in 2024 combining mostly-local attention with KV sharing across layers, reporting a large reduction in cache size — a reminder that the cache can be attacked from several directions at once.
• State space plus attention plus MoE. Jamba (AI21, 2024) interleaves Mamba layers with attention layers at a documented ratio of one attention layer per block of eight, and adds MoE feed-forward blocks. NVIDIA’s Nemotron-H (2025) is a Mamba-transformer hybrid in the same spirit.
• Grouped-query and multi-query attention. Now near universal, and worth counting as a hybrid technique: sharing K and V across query heads reduces kv_heads in the formula above directly, typically by four to eight times, at a small quality cost.

Reading a model card for the mix

1. Find the layer count and the KV head count. If queries and keys have different head counts, the model uses grouped-query attention; the ratio tells you the saving.
2. Look for a window size. A stated sliding window means the cache is capped per layer. Check whether it applies to all layers or alternates.
3. Count the global layers. Hybrid cards usually say how many layers are full-attention. That number times 2 * kv_heads * head_dim * bytes is the cache growth per token — the number that decides your concurrency.
4. Check for total versus active parameters. Two figures means an MoE feed-forward block, and the smaller one drives cost per token.
5. Treat the advertised context as an upper bound. A long window is a claim about what fits, not about what the model uses well, and the two diverge more in hybrids because most layers are compressing. Effective context length is the property that matters.

What hybrids cost

• Quality is not predictable from the ratio. There is no formula from the mix to capability. The ratio is found by training, which means a hybrid is a more expensive design to get right than a uniform stack.
• Kernels and serving support lag. Every layer type needs its own optimised kernel, its own quantisation path and its own support in whichever inference server you use. A model that mixes three types needs all three to be mature at once.
• Recall degrades unevenly. A hybrid can be excellent on most long-context work and fail on a specific retrieval pattern that happens to fall between its attention layers. That is harder to characterise than a uniform weakness, and it makes evaluation on your own workload more necessary, not less.
• Two memory profiles in one model. Part of the footprint grows with context and part does not, which makes capacity planning less linear than for a pure transformer.

The trade, one last time: a hybrid buys most of attention’s recall at a fraction of attention’s memory, and pays with complexity — in the kernels, in the serving stack, and in a quality profile that has to be measured rather than derived. The fact that this is where nearly everyone landed, rather than at either extreme, is the most informative result in this cluster.