Splitting and Merging Multi-Part GGUF Files

A multi-part GGUF is not one file cut into pieces. Each shard is a complete, valid GGUF with its own header and its own metadata, plus three extra keys saying which shard it is. That is why concatenating them produces garbage and why the split tool exists.

These are not pieces of one file

People reach for cat model-*.gguf > model.gguf because the filenames look like the output of split. They are not. Every shard begins with the GGUF magic bytes and a full key-value metadata block, so concatenation produces a file whose second header sits in the middle of what the loader thinks is tensor data.

What distinguishes a shard from a standalone model is three metadata keys that llama.cpp writes into each part: split.no, the zero-based index of this shard; split.count, how many there are; and split.tensors.count, the total tensor count across the whole model. The loader reads those from the first file and uses them to find and validate the rest. The filenames follow a fixed pattern — prefix-00001-of-00003.gguf — and the loader derives the sibling names from it, which is why renaming a shard breaks loading even though nothing inside the file changed.

Splitting

The tool is llama-gguf-split, built with the rest of llama.cpp. Splitting is the default mode. Two ways to say how big the pieces should be:

# by size: no shard larger than 4 GB
./llama-gguf-split --split --split-max-size 4G \
  model-Q4_K_M.gguf model-Q4_K_M-split

# by tensor count (the documented default is 128 tensors per shard)
./llama-gguf-split --split --split-max-tensors 64 \
  model-Q4_K_M.gguf model-Q4_K_M-split

# see the plan without writing anything
./llama-gguf-split --split --split-max-size 4G --dry-run \
  model-Q4_K_M.gguf model-Q4_K_M-split

The second positional argument is a prefix, not a filename: the tool appends the index suffix itself and writes model-Q4_K_M-split-00001-of-00003.gguf and siblings. Run the dry-run first on anything large; it prints the split plan and exits, which is much cheaper than discovering the shard boundaries you wanted were not the ones you got.

One option worth knowing is --no-tensor-first-split, which keeps the first shard free of tensor data. That gives a small first file holding only metadata — convenient when a consumer wants to read the architecture, the context length and the quantization type without fetching gigabytes. See llama.cpp’s gguf-split README for the current option list.

Loading shards without merging

Usually you do not need to merge at all. Point -m at the first shard and llama.cpp finds the rest:

./llama-cli -m ./model-Q4_K_M-split-00001-of-00003.gguf -p "hello" -n 32

All three files must be in one directory with their original names. If one is missing or truncated the loader fails at load rather than producing wrong output, because split.tensors.count gives it an expected total to check against — a genuinely useful property of the design, and one you can inspect directly when a model will not load.

Merging back

1. Put every shard in one directory under its original name.
2. Run ./llama-gguf-split --merge model-Q4_K_M-split-00001-of-00003.gguf model-Q4_K_M-merged.gguf. You name only the first shard; the tool locates the others from the metadata and the naming pattern.
3. Confirm the merged file loads, and that its tensor count matches the split.tensors.count the shards carried.
4. Only then delete the shards. The tool offers --delete-splits to remove them as part of the merge; leaving it off until you have verified the output is the cheaper habit.

A merge needs free space for the whole model on top of the shards, so budget roughly twice the model size on that filesystem. On a machine where that is the problem in the first place, loading from shards directly is the better answer.

Choosing a shard size

Shard size is a distribution decision, not a performance one — the loader memory-maps the tensor data either way, so the boundaries do not cost inference time. What they do interact with is whatever is moving the file:

• Filesystem limits. FAT32 caps a single file at 4 GB, which is why 4G is such a common choice for anything that might touch a USB stick or an SD card.
• Resumable transfer. A failed download of a 45 GB file restarts a 45 GB download. Ten shards mean losing one shard.
• Parallel fetch. Several shards can be pulled at once from a store that limits per-connection throughput.
• Hosting limits. Model registries impose per-file size caps, and those caps are the usual reason a large quantization is published pre-split.

To size the shards you need the model’s total bytes, which is a derivation rather than a guess: a pure Q4_K tensor costs 4.5 bits per weight by llama.cpp’s own documentation, so a 70B model at that rate is 70e9 × 4.5 / 8 ≈ 39.4 GB of weights before metadata. Real Q4_K_M files run higher because the M variant keeps some tensors at 6 bits — check the file size rather than trusting the formula for mixed schemes. The general version of that arithmetic is on the sharding math page.

When a shard set will not load

Four causes account for nearly all of it, and they are distinguishable from the filenames and the metadata alone.

• A renamed file. The loader derives sibling names from the pattern, so model-part2.gguf is invisible to it even if the bytes are perfect. Restore the -00002-of-00003.gguf form exactly, including the five-digit zero padding.
• A missing or truncated shard. A download that stopped early leaves a file of the right name and the wrong length. Because each shard records split.tensors.count for the whole model, the loader can tell that tensors are absent and will say so rather than running on a partial model. Compare file sizes against whatever checksums the source published.
• Shards from two different builds. Two quantizations of one model produce shard sets with identical filenames. Mix them in one directory and the second set silently overwrites the first, which gives you a complete-looking set whose parts do not belong together. Keep each set in its own directory.
• Concatenated shards. If somebody has already run cat over them, the result is not recoverable by splitting it again — the header boundaries are now interior bytes. Re-fetch.

Two habits avoid most of this. Keep a shard set in its own directory with nothing else in it, and verify a merge before deleting the parts rather than trusting an exit code. Neither costs anything, and both address failures that otherwise present as a corrupt model rather than as a file-management mistake.