How Big a Model Is on Disk: Parameters Times Bytes Per Weight

A model file’s size is its parameter count times the number of bytes each parameter is stored in. That is the entire calculation, and the reason it still surprises people is that the number it produces is not the amount of memory you need to run the thing.

The rule

  file_size_bytes  =  parameters  x  bytes_per_weight

  FP32   4 bytes per weight
  FP16   2 bytes per weight     (half precision)
  BF16   2 bytes per weight     (brain float, the usual release format)
  FP8    1 byte  per weight
  INT8   1 byte  per weight
  INT4   0.5 bytes per weight

Worked for a 7-billion-parameter model at BF16:

  7e9 * 2  =  1.4e10 bytes  =  14 GB

The parameter count includes the embedding and output-projection matrices, which for a small model with a large vocabulary are a non-trivial share of the total. A 200,000-entry vocabulary with a width of 4,096 is 819 million parameters in the embedding table alone — more than a tenth of a 7B model. That is why small models have grown less quickly than vocabularies have.

Every precision, worked

Model size, in GB (decimal, 1 GB = 1e9 bytes):

  params    FP32     BF16      FP8     INT4
  ------  -------  -------  -------  -------
     1B     4.0      2.0      1.0      0.5
     3B    12.0      6.0      3.0      1.5
     7B    28.0     14.0      7.0      3.5
     8B    32.0     16.0      8.0      4.0
    13B    52.0     26.0     13.0      6.5
    32B   128.0     64.0     32.0     16.0
    70B   280.0    140.0     70.0     35.0
   405B  1620.0    810.0    405.0    202.5

Every cell is just parameters x bytes_per_weight. Nothing else is
happening.

Read across a row and the practical consequences fall out immediately. A 70B model at BF16 needs 140 GB of memory for weights alone, which no single accelerator of the current generation provides, so it must be split across at least two — see multi-GPU inference. The same model at INT4 is 35 GB and fits comfortably on one 80 GB part with room for a cache.

Read down a column and you get the other consequence: quantisation is the single largest lever on whether a model runs at all, which is why choosing a quantisation is usually the first decision rather than the last.

GB against GiB, and where the 7% goes

Model files are described in decimal gigabytes and operating systems report binary gibibytes, and the difference is large enough to matter when you are deciding whether something fits.

  1 GB  = 1,000,000,000 bytes        (decimal, SI)
  1 GiB = 1,073,741,824 bytes        (binary, 2^30)

  ratio = 1.073741824,  a 7.37% difference

A 70B model at BF16:
  1.4e11 bytes  =  140.0 GB  =  130.4 GiB

An 80 GB accelerator:
  the "80 GB" is decimal in the marketing and the usable figure after
  driver and framework reservation is lower still — typically a few GB
  less than the nameplate.

So "70 GB of weights on an 80 GB card" is genuinely tight rather than
comfortably fine, and this is exactly where out-of-memory errors come
from at load time rather than at inference time.

Why a 4-bit file is not 4 bits per weight

Quantised distributions in practice use mixed precision: important tensors are kept at higher precision, and quantisation carries per-block scaling factors that occupy space themselves. So the effective bits per weight is above the nominal figure, usually by ten to thirty per cent.

  file_size_bytes  =  parameters  x  bits_per_weight / 8

For a 7B model:
  nominal 4.00 bits/weight:  7e9 * 4.00 / 8  =  3.50 GB
  actual  4.80 bits/weight:  7e9 * 4.80 / 8  =  4.20 GB
  actual  5.50 bits/weight:  7e9 * 5.50 / 8  =  4.81 GB

To recover the real figure for any file you have, invert it:

  bits_per_weight  =  file_size_bytes * 8 / parameters

A 4.37 GB file for a 7B model:
  4.37e9 * 8 / 7e9  =  5.0 bits per weight

That inversion is the useful trick: it lets you compare two quantised releases of the same model on the only axis that matters, which is how much information per weight survived. Quality effects of the choice are covered in inference quantisation and the transparency problem in quantised model transparency.

The KV cache, derived

The weights are constant. The KV cache grows with every token and with every concurrent request, and it is what actually decides how many users a machine can serve.

  kv_bytes_per_token  =  2 * n_layers * n_kv_heads * head_dim * bytes

  the 2 is for K and V
  n_kv_heads is the *key/value* head count, which under grouped-query
  attention is much smaller than the attention head count

Worked for a 70B-shaped model with grouped-query attention:
  n_layers = 80, n_kv_heads = 8, head_dim = 128, FP16 (2 bytes)

  2 * 80 * 8 * 128 * 2  =  327,680 bytes per token  =  320 KiB/token

At a 32,768-token context, one sequence:
  32,768 * 327,680  =  1.07e10 bytes  =  10.7 GB

At batch 16, same context:
  16 * 10.7 GB  =  172 GB

The cache is now larger than the 140 GB of BF16 weights it serves.

Now run the same model without grouped-query attention, where every attention head has its own key and value heads — 64 of them instead of 8:

  2 * 80 * 64 * 128 * 2  =  2,621,440 bytes per token  =  2.5 MiB/token

At 32,768 tokens, one single sequence:
  32,768 * 2,621,440  =  8.59e10 bytes  =  86 GB

One user, one conversation, 86 GB of cache.

That factor of eight is the entire commercial reason grouped-query
attention exists, and it is why long context became affordable at all.

Two further reductions are in common use and both are visible in this formula. Quantising the cache to 8 bits halves the bytes term. And multi-head latent attention, used by some recent architectures, restructures what is cached rather than shrinking the head count.

What actually has to fit

  total_memory  =  weights
                +  kv_cache (per sequence x concurrent sequences)
                +  activations (transient, scales with batch and width)
                +  framework and driver overhead (typically 1-2 GB)
                +  fragmentation headroom

Worked: a 70B model at INT4, 8 concurrent users, 8k context each,
with an FP16 KV cache at 320 KiB per token:

  weights      70e9 * 0.5                    =  35.0 GB
  kv cache     8 * 8192 * 327,680            =  21.5 GB
  overhead     framework + activations       ~   3.0 GB
                                             ------------
  total                                      ~  59.5 GB

Fits on an 80 GB accelerator with headroom.
Raise the context to 32k and the cache alone becomes 86 GB, and it
does not.

The lesson that keeps catching people: the model loaded fine and the server fell over an hour later. Weights are allocated at load and the cache grows with traffic, so a configuration that starts cleanly can run out of memory under concurrency. Size for the cache at your intended context and batch, not for the file. There is a fuller treatment in VRAM requirements.

None of the arithmetic on this page can go out of date, because none of it depends on a price, a product or a vendor. Substitute the parameter count, layer count and head configuration from any model card and the answers follow.