The Math Behind Splitting a Model Checkpoint Into Shards

Shard count looks like a division and is not. The division gives a lower bound; the actual count comes from a greedy packing that never splits a tensor and never reorders one, and the gap between the two is where every surprising shard layout comes from.

The three inputs

Everything below needs exactly three numbers, and all three are available without downloading any weights.

• Total bytes on disk, written T. For an existing sharded checkpoint you do not have to derive this: the metadata.total_size field in model.safetensors.index.json states it directly. For a model you are about to save, T = P × b, where P is the parameter count and b is bytes per parameter — 2 for fp16 or bf16, 4 for fp32.
• The cap, written C. This is max_shard_size. Two defaults are in play and they differ: the huggingface_hub serialization reference documents "5GB" for save_torch_state_dict and split_torch_state_dict_into_shards, while Transformers’ save_pretrained currently documents "50GB". Which applies depends on which function is writing your files.
• The largest single tensor, written M. Usually the token embedding: vocabulary size × hidden size × b. This is the input people forget, and it is the one that makes the arithmetic interesting.

The floor, and why it is only a floor

The naive answer is ceil(T / C). It is a genuine lower bound: you cannot fit T bytes into fewer shards when no shard may exceed C. But it is reached only when tensors happen to pack perfectly, which they do not.

Two documented rules push the real count above it. First, a tensor is never split across shards, so if a single tensor exceeds C it gets a shard to itself and that shard is larger than the cap — the huggingface_hub documentation says so explicitly. Second, and more often the cause, the packer is greedy over the state dict in key order with no attempt to fill shards optimally. Its own documentation gives the example: with a 10 GB limit and tensors of [6, 6, 2, 6, 2, 2] GB, the result is [6], [6+2], [6+2+2] — three shards where a bin-packer would have found three of a much more even shape, and where the floor is ceil(24/10) = 3.

One unit detail matters when you check the arithmetic. The parser is base ten: KB, MB, GB and TB mean powers of 1000, so "5GB" is 5,000,000,000 bytes, not 5,368,709,120. Comparing against what your file manager calls “GB” will make shards look about 7% over the cap when they are not.

The greedy packing that decides the real answer

The algorithm, stated plainly: walk the tensors in state-dict key order; keep a running total; when adding the next tensor would exceed C, close the current shard and start a new one with that tensor. Formally, an upper bound on the count is

floor      = ceil(T / C)
worst case = (number of tensors larger than C)
           + ceil((T - bytes in those tensors) / (C - M_small))

where M_small is the largest tensor that still fits under C.
The (C - M_small) term is the space a shard may be forced to leave empty
because the next tensor in key order would not fit.

In practice the count lands one above the floor far more often than two above, because most tensors in a transformer are small relative to a multi-gigabyte cap and only the embeddings are large. The waste is concentrated at the boundaries where an embedding-sized tensor refuses to fit into the tail of a shard.

One asymmetry is worth naming because it catches people converting between precisions. The cap is compared against bytes on disk, not against parameters, so halving the precision halves T and can halve the shard count — the same model saved in fp32 rather than bf16 doubles T and roughly doubles the number of files, with no change to the model at all. If a checkpoint has more shards than you expected, checking the dtype in the index before checking anything else is usually the fastest route to the answer.

The naming, once the count is known, is fixed: model-00001-of-00004.safetensors and so on, with both numbers zero-padded to five digits and the index starting at one. That total in the filename is why you cannot add a shard to an existing checkpoint without rewriting every name and the index.

A worked example

Assume an 8.03 billion parameter model saved in bf16, a 5 GB cap, and the embedding shape you get from a 128,256-token vocabulary at hidden size 4096 with an untied output head. Every number below follows from those assumptions and nothing else.

b = 2 bytes (bf16)
T = 8.03e9 * 2                = 16.06e9 bytes
C = "5GB"                     =  5.00e9 bytes   (base ten)

floor = ceil(16.06e9 / 5.00e9) = ceil(3.21) = 4 shards

largest tensors:
  embed_tokens  128256 * 4096 * 2 = 1.05e9 bytes
  lm_head       128256 * 4096 * 2 = 1.05e9 bytes   (untied)

M = 1.05e9 < C, so no tensor needs a shard of its own.
Waste per boundary is at most one tensor, <= 1.05e9 bytes,
so the packing can lose at most ~21% of one shard at each of
3 boundaries -> the count stays at 4 unless T sits within
~3.15e9 bytes of the next multiple of C. It does not (16.06 vs 15.00).

per-shard average = 16.06e9 / 4 = 4.02e9 bytes

Four shards averaging just over 4 GB each, none at the 5 GB cap. That last part surprises people who expect three full shards and a small one; greedy packing in key order produces roughly even shards when the tensors are small and uniform, which transformer layers mostly are.

Checking against a real repository

You do not have to trust any of this. The index file makes it checkable, and the check costs one small download:

1. Fetch model.safetensors.index.json from the model repository. It is a few hundred kilobytes at most.
2. Read metadata.total_size. That is T, measured rather than derived.
3. Count the distinct values in weight_map. That is the actual shard count.
4. Compare against ceil(T / C) for the cap you believe was used. A result one above the floor is the greedy packer; a result far above means a different cap than you assumed.
5. For a GGUF instead, read split.count out of the first shard's metadata — inspecting GGUF metadata gets you there without downloading tensor data.

The reason to do this rather than assume is that T from the index is the only number in the chain that is not an estimate. Parameter counts quoted in model names are rounded — “8B” covers anything from 7.5 to 8.5 billion — and a 6% error in P is enough to move the shard count by one when T/C lands near an integer.