Model Parameters: What 7B, 70B and 400B Actually Buy You

Parameter count is the most quoted number about a language model and the least useful one for deciding whether it can do your job. It is nearly perfect at predicting something else, and that something else is worth knowing precisely.

What a parameter is

A parameter is one learned number in one of the model’s matrices — an entry in an attention projection, a feed-forward weight, an embedding row. “7B” means about seven billion such numbers, and adding up the matrices of a published 7B model gets you to 6.74 billion exactly. It is a size, in the same sense that a binary has a size.

The part it predicts exactly

Weights first. Bytes per parameter depends on the numeric format: 4 for fp32, 2 for fp16 or bf16, 1 for 8-bit, about 0.5 for 4-bit. So 7B at bf16 is roughly 14 GB and 70B at bf16 is roughly 140 GB — the second does not fit on one 80 GB card, which is the first hard fact a parameter count gives you.

Then the KV cache, which people forget and which is what actually decides how many concurrent requests fit. Per token, per layer, you store one key and one value vector:

bytes/token = 2 (K and V) x n_layers x n_kv_heads x head_dim x bytes_per_value

Llama 2 7B, bf16, multi-head (n_kv_heads = n_heads = 32):
  2 x 32 x 32 x 128 x 2 = 524,288 bytes = 0.5 MB per token

  4,096-token context  ->  2.0 GB for ONE request
  32 concurrent requests at 4k  ->  64 GB, on top of the 14 GB of weights

That arithmetic is why grouped-query attention exists. Cut the number of key/value heads from 32 to 8 and the same line gives 0.125 MB per token — a quarter of the memory, with the query heads unchanged. It is also why a provider’s advertised context length and its concurrency limit are the same conversation.

Finally, arithmetic. The standard approximation is two FLOPs per parameter per token for a forward pass, so per-token compute tracks the count that is active, not the count on the box — the entire subject of mixture of experts, where a 400B model can do the work of a 40B one.

The part it predicts loosely

Scaling-law work relates loss to parameters as a power law: loss falls by a roughly constant amount for each constant multiple of parameters. Two things follow that are easy to misread. Going from 7B to 70B is one such step, and going from 70B to 700B is the same size of step — not ten times bigger. And loss is not usefulness. A drop in next-token loss can show up as fluency, or as a task crossing a threshold from unusable to usable, or as nothing you can perceive.

There is also a systematic reason parameter count reads high on capability in casual comparison: bigger models are usually newer, trained on more and better-filtered data, and post-trained harder. When a 70B release beats a 7B release from two years earlier, the parameter count is one of four things that changed, and it is not obviously the one doing the work. The comparison that isolates the variable — same family, same data recipe, same date, two sizes — is one only the publisher can run, and they usually do publish it.

Five places the correlation breaks

• Training tokens. An under-trained large model loses to a well-trained small one. This is the whole content of the Chinchilla result and of the overtraining that followed it. Parameters without data is capacity without content.
• Post-training. Instruction tuning and preference optimisation change how useful a model is with no change to N at all. The gap between a base model and its instruct sibling is enormous and invisible in the parameter count.
• Sparsity. Total and active counts diverge, and only one of them predicts price. Comparing a sparse model’s total against a dense model’s total compares nothing.
• Distillation. A small model trained on a large model’s outputs inherits behaviour its size would not have produced from scratch. Size stops being a proxy for training budget.
• Task shape. If the answer is in the context, the binding constraint is retrieval over that context, not parametric knowledge. Doubling parameters does not help a model find a clause it was given.

Reading a model card

The fields worth extracting, in order of how much they change a decision: active parameters if the model is sparse; total parameters; training tokens if disclosed; context length; whether the weights you are being served are quantized, which changes bytes and behaviour but not the parameter count; and whether the checkpoint is base, instruct or chat.

Notice that three of those six fields are absent from most announcements. Training tokens are disclosed inconsistently, served quantization is disclosed rarely, and active parameters are given only when the publisher is proud of them. The number that is always present is the one this page has spent its length qualifying — which is a reasonable summary of why it is quoted so much.