Importing a GGUF File Into Ollama

Ollama’s library is a curated subset of what exists. Sooner or later you want a fine-tune, a quantization or a language-specific model that is only published as a GGUF on Hugging Face, and the import is three lines — plus one step almost every walkthrough leaves out, which is the reason imported models so often answer strangely.

When you need this

Three cases. A model not in Ollama’s library at all, which is most community fine-tunes. A quantization the library does not carry — the library typically ships a 4-bit K-quant and an 8-bit build, and if you want a 5-bit or 6-bit variant to fit a particular card, somebody else has published it and Ollama has not. Or a GGUF you produced yourself by converting Safetensors with llama.cpp’s convert_hf_to_gguf.py, which is the path for a private fine-tune. See choosing a quantization for which of those variants is worth the trouble.

What you are importing is a single self-contained file. GGUF holds the tensors and a metadata block describing architecture, layer count, trained context length and tokenizer, which is why Ollama can serve a file it has never seen before without being told anything about it. What GGUF does not reliably carry is a chat template, and that gap is the whole of the last section.

Getting a file you are allowed to run

Weights carry licences, and a GGUF republished by a third party carries the original model’s terms rather than the republisher’s. Some base models are gated: access requires accepting terms on the publisher’s Hugging Face page and authenticating your download. If a model is gated, that is a condition of use, not an obstacle to route around — request access, accept the terms, and download with your own credentials. A community GGUF that exists only because it evades a gate is not a file to build a product on.

Check three things about a candidate file before downloading several gigabytes. The quantization suffix in the filename tells you the size and roughly the quality cost. The publisher’s model card should name the base model and the conversion tool, and one that names neither is unverifiable. And the file size should be consistent with the parameter count and the quantization — an 8B model at a 4-bit K-quant is around five gigabytes, and a file far off that is not what it claims.

The Modelfile

Ollama’s import documentation gives the minimum form: a FROM instruction pointing at the file.

FROM ./Qwen3-8B-Q5_K_M.gguf

The path is resolved as an absolute path, or relative to the Modelfile’s own location — not relative to the directory you happen to be standing in. Putting the Modelfile next to the GGUF and building from there removes the ambiguity entirely.

In practice you want more than the minimum, because a bare GGUF import inherits no parameters and no system message either:

FROM ./Qwen3-8B-Q5_K_M.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
{{ .Response }}<|im_end|>
"""

PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
PARAMETER temperature 0.6
PARAMETER num_ctx 16384

That template is the ChatML format, which is what Qwen-family models expect; a Llama-family model wants <|start_header_id|> markers instead, and a Gemma model wants something else again. The template must match the model, not your preference. The variables available are {{ .System }}, {{ .Prompt }} and {{ .Response }}, covered in Ollama’s TEMPLATE field.

Build it and run it

1. Put the GGUF and a file named Modelfile in the same empty directory, and change into it.
2. Build: ollama create qwen3-q5 -f Modelfile. Ollama reads the GGUF, writes it into the blob store as a model layer, writes the template and params as their own layers, and finishes with writing manifest then success. Expect the build to take as long as copying the file, because that is largely what it is.
3. Confirm what was read out of the file, not what you hoped: ollama show qwen3-q5 prints the architecture, parameter count and quantization level that came from the GGUF metadata. If the parameter count is not what you expected, you downloaded a different model.
4. Run it: ollama run qwen3-q5, and ask two or three questions with different shapes — a factual one, a multi-turn one, and one that should make it stop cleanly.
5. Once it works, the original GGUF is redundant: the weights now live in the blob store as their own copy. Deleting the download frees the duplicate, and rebuilding later means downloading again.

Disk use is worth planning for, because an import temporarily needs the file twice: once as the download and once as the blob Ollama writes. On a machine with 20 GB free, importing a 15 GB fp16 GGUF fails at the copy rather than at the build, and the error arrives after the slowest part of the operation.

When the output looks wrong

The characteristic symptoms of a template mismatch are specific enough to diagnose from the output alone. The model emits its own special tokens as visible text — <|im_end|> or <|eot_id|> appearing in the answer — which means the tokens are not in the stop list. It keeps going after answering and starts writing your side of the conversation, which is the same problem. It ignores the system message entirely, which means the template has no {{ .System }} branch. Or it answers a chat-style question as though continuing a document, which means no template is being applied at all.

The reliable fix is to copy the template from a model of the same family that Ollama already ships. ollama show --template qwen3 prints a working ChatML template; ollama show --parameters qwen3 prints the stop list that goes with it. Paste both into your Modelfile, rebuild, and the imported file behaves like the library version because it now is configured like the library version.

One thing an import cannot fix is the context ceiling. The trained context length is metadata inside the GGUF, and asking for a larger num_ctx than the file declares gets you the file’s value, clamped without complaint — see what num_ctx actually sets.