RWKV and Recurrent Models With Transformer Quality

RWKV is an architecture with two equivalent forms: a parallel form used for training, and a strict recurrent form used for generation, where the memory per sequence is a fixed number of vectors that does not grow with the context. It is the same goal as Mamba, reached from linear attention rather than from state space models.

The idea: two forms of one layer

Standard attention computes a softmax over the dot products of one query against every key. The exp inside that softmax is applied to the query-key product, which is what couples every pair of positions and produces the quadratic cost.

Linear-attention variants move the nonlinearity so that it applies to keys and queries separately, before they meet. Once that is true the sum over previous positions can be accumulated incrementally, because it is just a running total. RWKV takes that structure and adds a learned per-channel decay, so older contributions fade rather than being weighted purely by content.

The name is its four learned projections: Receptance (a gate on what is read out), Weight (the decay, learned per channel), Key and Value. Only the last two mean what they mean in attention.

What the WKV operator computes

In its summed form, the output at position t is a weighted average of all previous values, where the weight of a value from i steps back is its key’s exponential times a decay raised to i:

             sum over i<t of  exp(-(t-1-i)*w + k_i) * v_i
wkv_t  =  ------------------------------------------------
             sum over i<t of  exp(-(t-1-i)*w + k_i)

w is a learned positive number per channel: the decay rate.

Written that way it looks like an O(T) sum per position. Written recurrently it is two running accumulators, a numerator and a denominator, updated once per token:

decay = exp(-w)

a_t = decay * a_{t-1} + exp(k_t) * v_t        numerator
b_t = decay * b_{t-1} + exp(k_t)              denominator

wkv_t  = a_t / b_t
out_t  = sigmoid(r_t) * (W_out @ wkv_t)       receptance gates the read

Two vectors per channel group, updated with a multiply and an add. That is the whole memory of the layer. In real implementations the exponentials are kept in a running-maximum form to avoid overflowing — the same stabilisation a softmax uses — which adds a third accumulator but does not change the shape of the argument.

The dividing line against attention is visible in that recurrence. A transformer keeps k_i and v_i for every i and can go back and read any of them. RWKV has already folded them into a and b, and no later query can separate them again.

The state, in bytes

Take a 7B-class configuration: model dimension 4,096, 32 layers, 16-bit. Per layer the recurrent state is the time-mixing accumulators plus the shifted token from the previous step in each of the two sub-blocks. Counting generously at five vectors of the model dimension per layer:

state per layer = 5 * 4,096 values = 20,480
                = 40,960 bytes at fp16      (40 KB)

state per model = 40 KB * 32 layers = 1.3 MB per sequence
                  ...constant in sequence length.

Now the same model as a transformer, with 32 heads of dimension 128 and no grouped-query sharing:

bytes per token = 2 * 32 layers * 32 heads * 128 * 2 bytes
                = 524,288 bytes    (512 KB per token)

  2,048 tokens ->  1.0 GB
 32,768 tokens -> 16.8 GB

The crossover is at three tokens. Everything after that is the recurrent model’s margin, and it grows without bound. The practical consequence is not speed but concurrency and reach: a fixed 1.3 MB per session means the number of sessions a box can hold is set by the model weights alone, and it means a context of a million tokens costs no more memory than a context of a thousand.

Token shift and channel mixing

Two smaller mechanisms do more work here than their size suggests.

Token shift interpolates each input with the previous token’s input before the projections: x’ = mu * x_t + (1 - mu) * x_{t-1}, with mu learned per channel. It costs one extra vector of state and gives every channel a cheap two-token window, which is enough for the layer to form bigram-like features without touching the recurrence.

Channel mixing is RWKV’s feed-forward block: the same token-shift trick, a squared-ReLU projection up and back down, and a receptance gate. It plays the role the MLP plays in a transformer block, and it holds most of the parameters.

Later versions changed the decay from a fixed learned scalar per channel to something data-dependent, which is the same move Mamba made when it went from S4 to selective state: version 5 (Eagle) and version 6 (Finch) in 2024 introduced matrix-valued state and dynamic decay, and version 7 (Goose) in 2025 continued it. The direction of travel in both families is identical — make the forgetting depend on the content.

Prefill, decode, and where each form is used

The two forms are not a choice made once. A serving stack uses both inside a single request, and knowing which is running when is what stops the constant-memory claim being overread.

1. Prefill. The prompt is known in full, so the sequence-level form processes all its positions together and produces one thing: the state after the last prompt token. This is compute-bound and parallel, exactly like a transformer’s prefill.
2. Decode. From that state, the recurrence runs one step per generated token. Each step reads the model weights, which is also exactly like a transformer’s decode.

So the honest statement is narrow. Per token, at short context, an RWKV model is not inherently faster than a transformer of the same size, because both are dominated by reading the weights. What changes is the second term. A transformer’s decode step also reads a cache that grows with every token; RWKV’s reads a state that does not. The advantage therefore appears precisely where the cache is the binding constraint — long contexts, many concurrent sessions, or memory-limited hardware — and is close to nothing on a short prompt.

One operational consequence is genuinely distinctive. A session is 1.3 MB of state, so it can be serialised, stored, and resumed later on a different machine at full fidelity. The equivalent for a transformer is gigabytes of cache or a complete re-prefill of the conversation. For long-lived assistants and for anything that suspends and resumes, that is a difference in kind rather than in degree.

What constant memory costs

Honest status

RWKV is a real, open, community-led architecture with released weights across several generations and sizes into the low tens of billions of parameters, and it is used in production in places where the constant state is the deciding property — long-running sessions, embedded and CPU inference, and very long inputs on constrained memory.

It has not displaced attention at the frontier, and nothing public establishes that it matches the strongest attention models at equal training compute. The pattern that has actually taken hold in shipped models is the same one state space models ran into: mix a few attention layers in and keep most of the memory saving. If you are choosing an architecture rather than studying one, that is the version to look at first.