Skip to content
Multigrid

Model files, adapters and conversion

GGUF, safetensors, ONNX and LoRA adapters — converting between them, checking what you downloaded, and which conversions do not exist.

A model on disk is a container format wrapped around a particular way of storing numbers, and almost every frustrating hour in local inference comes from confusing the two. GGUF, safetensors and ONNX are containers: they say where a tensor starts, what shape it is and what it is called. Q4_K, GPTQ, AWQ and bf16 are storage schemes: they say what the bytes in that tensor mean. A conversion that only changes the container is mechanical and lossless. A conversion that has to change the storage scheme is a re-quantization, and it can only be done well from weights that have not been quantized yet.

These pages work from that distinction outward. They cover loading and switching LoRA adapters in the runtimes that accept them, the arithmetic that decides how a checkpoint gets split into shards, the two exports most people need — safetensors and ONNX — and the conversions that look reasonable and do not exist, with the reason stated rather than the error repeated. Where a command line is given it is the documented one at the time of writing, and the mechanism underneath it is given too, because the flags move and the mechanism does not.

Loading a LoRA Adapter in llama.cpp on Top of a GGUF

Convert a PEFT adapter to GGUF, apply it to a quantized base at load time, and prove the output changed.

9 min read

Switching LoRA Adapters in Ollama

Two Modelfiles over one base with different ADAPTER weights, and what Ollama actually does with them at create time.

8 min read

Applying a LoRA at Inference Time Without Merging It

What the extra low-rank matmul costs in arithmetic and memory, derived from rank and layer dimensions.

9 min read

LoRA Adapter Compatibility Across Quantization Formats

Which adapter file each local runtime accepts, and why the adapter's format is bound to the loader rather than to the base's quantization.

9 min read

Testing Whether a LoRA Adapter Is Actually Doing Anything

A differential test that separates a loaded adapter from an applied one, for the failure that prints no error.

9 min read

Why GPTQ Weights Usually Cannot Be Converted Directly to GGUF

The tensor-layout mismatch behind the conversion error, and why the fix is to start from the original fp16 weights.

9 min read

Converting a PyTorch Checkpoint to Safetensors

Turning a .bin checkpoint into .safetensors, verifying every tensor matches, and handling the shared-memory error.

9 min read

Splitting and Merging Multi-Part GGUF Files

Using llama.cpp's gguf-split to shard a large GGUF and put it back, and why cat does not work.

8 min read

The Math Behind Splitting a Model Checkpoint Into Shards

Deriving shard count and per-shard size from parameter bytes and a size cap, and why the answer is a lower bound.

9 min read

Converting a Hugging Face Model to ONNX

Exporting a transformers model with optimum-cli, reading the validation output, and handling models over 2 GB.

9 min read

Extracting a LoRA Adapter's Weights From an Already-Merged Model

Why recovering an adapter needs the original base to subtract, and why the result is an approximation rather than the original.

9 min read

Checking a GGUF File's Metadata Before You Trust It

Read a GGUF's architecture, training context length, quantization mixture and tokenizer straight out of its header, without loading a single tensor.

9 min read

Why a Model's File Size Never Exactly Matches Its Parameter Count

Where the gap between bytes-per-parameter arithmetic and the number your file manager reports actually comes from, worked through with published figures.

10 min read

Verifying a Local LLM Setup Makes Zero Network Calls

Four ways to demonstrate that a local model stack is not talking to anything, from weakest to strongest, ending in a test that cannot give a false pass.

10 min read

Local Inference as Part of a HIPAA Compliance Strategy

Running the model on your own hardware removes one disclosure and none of the Security Rule; here is which obligations it touches and which it leaves entirely intact.

11 min read

What Local Inference Cannot Guarantee About Privacy

Local inference is a claim about one network hop, not about the life of your data; here is everything that hop leaves untouched.

10 min read

Setting Up a Home Lab for Local LLMs

Which hardware tier you actually need, derived from weight size and KV cache arithmetic rather than from a shopping list.

11 min read

Running a Local LLM on a NAS

Get a small quantized model serving from a Synology or Unraid box, with the memory-bandwidth ceiling worked out before you start so the result is not a surprise.

10 min read

Local LLMs for a Small Business Without a Cloud Budget

The break-even arithmetic for buying hardware instead of paying per token, including the two costs that usually decide it and never appear in the comparison.

11 min read

Getting New Model Weights Onto an Air-Gapped Machine

The checksum-verified physical-media workflow for updating a model that is already deployed behind an air gap, including the rollback you will need.

10 min read

Does Running a Model Locally Actually Stop It Training on Your Data

No gradient step happens during inference wherever it runs, which makes the local/hosted distinction irrelevant to this particular fear — and relevant to a different one.

9 min read

Other topics