DBRX’s Mixture-of-Experts Architecture and Context Window

Databricks released DBRX in March 2024 with two parameter counts on the card — 132 billion total, 36 billion active — and a 32K-token context window. The 36B figure is the one that governs price and latency, and it is produced by a routing choice that is more specific than “mixture of experts”.

The documented figures

From Databricks’ DBRX model card and its accompanying technical blog post at databricks.com:

Total parameters        132B
Active per token         36B
Experts                   16
Experts used per token     4
Context window        32,768 tokens
Pretraining data          12T tokens of text and code
Attention                 grouped-query attention
Position encoding         rotary (RoPE)
Tokenizer                 GPT-4's tiktoken BPE vocabulary

Two checkpoints were released: DBRX Base and DBRX Instruct. The figures above apply to both.

Read the first two lines as a pair. 132B is what you must fit in memory; 36B is what runs for each token. Those two numbers answer two different questions and neither substitutes for the other: the first decides whether you can host the model at all, the second decides what a token costs and how fast it arrives. A catalogue entry showing DBRX priced near a 30-something-billion-parameter dense model is not a mistake — it is the second number being reflected honestly.

What fine-grained routing means

The general mechanism is covered in why a 400B model can cost like a 40B one: a mixture-of-experts layer replaces one feed-forward block with many, and a router selects a few per token, so the model is large in memory and small in compute. DBRX’s specific contribution is in the numbers 16 and 4.

Contemporary open MoE models tended to use eight experts and select two. DBRX uses sixteen and selects four — smaller experts, more of them, more selected. The active parameter count works out similarly either way, because four smaller experts are about as much arithmetic as two larger ones. What differs is how many distinct combinations the router can express.

8 experts choose 2  →  C(8,2)  =   28 combinations
16 experts choose 4 →  C(16,4) = 1820 combinations

1820 / 28 = 65×

That is arithmetic you can check, not a benchmark claim: Databricks describes this sixty-five-fold increase in routing combinations as the motivation for the design. The argument is that a router with more expressible combinations can specialise more finely at the same inference cost. Whether it produces better output is a question about the trained model, not about the combinatorics, and the combinatorics alone do not settle it.

The same fine-grained approach appears in later sparse models — the DeepSeek V3 expert configuration and Qwen 3’s MoE sizes both push it further — so DBRX is a useful reference point for reading those cards.

There is a cost to the finer granularity and it is on the router, not on the experts. A router choosing four of sixteen has to produce a meaningful ranking over sixteen options for every token at every MoE layer, and if it does that badly the model both wastes capacity and becomes harder to batch, because a poorly calibrated router spreads tokens across more experts than necessary. Load balancing during training exists to stop a handful of experts absorbing most of the traffic while the rest sit unused — a failure mode that leaves you paying for 132B of weights to get the behaviour of a much smaller model. Whether a released checkpoint got that right is not visible from the card; it shows up as throughput variance under mixed traffic.

The 32K window and what limits it

DBRX uses rotary position embeddings and grouped-query attention, and was trained with a 32,768-token sequence length. Nothing about the MoE layers affects this: expert routing operates on the feed-forward position and the attention layers are conventional and dense. The context window is set by the attention stack and the training recipe, exactly as it would be in a dense transformer.

This is worth stating plainly because the two headline features of the card — sparse and 32K — get read as related. They are not. If you want a case where the architecture genuinely does change the context economics, that is the hybrid stack in Jamba, where most layers keep no KV cache at all. In DBRX, every one of the attention layers keeps a full cache and it grows with sequence length in the ordinary way.

Grouped-query attention does reduce that cache, by sharing key and value projections across groups of query heads, which is why almost every model of this generation uses it. It is a constant-factor saving, not a change of shape.

One more thing follows from the window being ordinary: everything you know about managing a 32K transformer applies unchanged. Long prompts push time-to-first-token up because prefill is quadratic in sequence length. Concurrency falls as contexts lengthen because cache is per-request. And a prompt that overflows produces a context-length error rather than silent truncation, in most serving stacks — which is the behaviour you want, and worth confirming on yours, because the alternative is a model quietly answering from half your prompt.

What it takes to serve

• Memory follows 132B, not 36B. Every expert must be resident. At 16-bit precision the weights alone are on the order of 264GB, so DBRX is a multi-GPU deployment before you have allocated a single token of KV cache. This is the standard sparse-model trade: you buy cheaper tokens with more expensive hardware.
• Arithmetic follows 36B. Latency per token and per-token price track the active count, which is why sparse models are priced far below what their total size suggests.
• Batching is less predictable. Which experts a token needs depends on the token, so a batch of diverse requests touches more experts than a batch of similar ones. Throughput under mixed traffic is harder to model than for a dense model of the same active size.

Licence and tokenizer

DBRX was released under the Databricks Open Model License, which is not Apache 2.0. It permits commercial use with an acceptable-use policy attached and a scale condition on very large deployments — the same category of open-weight-but-conditional licence discussed for Falcon’s licence history. If your compliance process distinguishes OSI-approved licences from conditional grants, DBRX is on the conditional side of that line, and the current text on Databricks’ site governs rather than any summary of it.

The tokenizer is notable for an unrelated reason: DBRX uses OpenAI’s tiktoken BPE vocabulary rather than a Databricks-specific one. That means token counts for DBRX line up closely with GPT-4-family counts and diverge from Llama or Mistral counts on the same text, which matters if you are estimating cost across providers from one tokenization. See why token counts differ between tokenizers before assuming a character-per-token ratio transfers.

The tokenizer choice also tells you something about the intended use. Adopting an existing widely deployed vocabulary rather than training one means giving up any gain a domain-tuned vocabulary would offer, in exchange for every tool that already counts GPT-4 tokens working unchanged. For a model released to be adopted rather than to win a benchmark, that is a reasonable trade, and it is the same reasoning that led 01.AI to ship Yi with a Llama-compatible architecture: compatibility with existing tooling is worth real points on a release nobody has integrated yet.