Running a Model With llamafile, No Install Required

There is no package to install and no runtime to configure. You download an executable that already contains the weights, mark it executable, and run it. On a reasonable connection the whole thing is two commands and a download.

Download one file

Mozilla publishes ready-made llamafiles on Hugging Face. The project’s own quickstart uses a small one, which is the right choice for a first run because it removes memory as a variable:

curl -LO https://huggingface.co/mozilla-ai/llamafile_0.10/resolve/main/Qwen3.5-0.8B-Q8_0.llamafile

That file is about 1.77 GB, and essentially all of it is weights: the runtime itself is a small fraction. Note the naming convention — model, quantisation, .llamafile — because it is the only documentation of what is inside once the file is on your disk. Bigger models exist in the same collections; read how the format works before you reach for one over 4 GB on Windows, where a platform limit applies.

Run it

1. macOS, Linux, BSD. Mark it executable and run it:

chmod +x Qwen3.5-0.8B-Q8_0.llamafile
./Qwen3.5-0.8B-Q8_0.llamafile

1. Windows. Rename the file to end in .exe and double-click or run it from a terminal. Windows will not execute it otherwise, because the extension is what tells it this is a program.
2. Either way. On the 0.10 line the file drops you straight into a chat interface in the terminal. Type a question and press enter; that is your first response, from one downloaded file with nothing else installed.

The terminal interface has its own commands rather than being a bare prompt: /help lists them and /upload attaches an image on models that accept one. Stop it with Control-C.

What appears before the prompt is a burst of startup output naming the model, the quantisation, the context size and the hardware backend it selected. That last line is the one to keep: it tells you whether a GPU was found. If it says CPU on a machine that has a GPU, the model will still answer and it will be much slower, and the cause is almost always the host’s drivers rather than llamafile — a portable file guarantees the code runs, not that your vendor’s compute stack is installed. Nothing is written outside the current directory, so if you decide against the whole thing, deleting the one file is a complete uninstall.

The web UI and the two APIs

While that process runs it is also serving HTTP on port 8080. Open http://localhost:8080/ for a browser chat interface over the same loaded model — the terminal chat and the web UI are two front ends onto one process, not two copies of the model.

The same port carries two API surfaces: an OpenAI-compatible one and an Anthropic Messages-compatible one. That second one is unusual among local runtimes and is genuinely useful, because a client written against Anthropic’s SDK normally has no local option at all. Point either SDK at the base URL and give it any non-empty key:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "one sentence on memory mapping"}],
    "max_tokens": 128
  }'

For streaming, both surfaces emit their provider’s own event shape, which is the thing to check before assuming a client will just work — see streaming responses. The two surfaces are the same model with two request and response schemas in front of it, so a difference in output between them is a difference in how your messages were assembled into a prompt, not a difference in what is doing the generating.

Your own GGUF with the bare runtime

The releases also publish the runtime on its own, without weights. Run that against any GGUF you already have:

./llamafile -m ~/models/qwen3-8b-q4_k_m.gguf -ngl 999

-ngl is the GPU layer count, and a large number means “all of them, as far as they fit”. Two release variants are worth knowing apart: the standard llamafile bundles GPU libraries and is larger, while llamafile-thin omits them. The same releases carry whisperfile and transcribefile for speech and diffusionfile for images, all built on the same portability machinery. This external-weights mode is also the documented answer to the Windows size limit.

Splitting the runtime from the weights changes the update story in a way that is worth planning for. A bundled llamafile pins one runtime to one model forever, which is exactly what you want when handing it to somebody else and exactly what you do not want on your own machine: a new release means re-downloading the weights you already have. Keep your GGUFs in one directory and update only the runtime, and the upgrade becomes a few hundred megabytes rather than a few gigabytes per model. The project’s zipalign tool goes the other way, packing a GGUF you already hold into a bare runtime to produce a single-file build of your own.

The three things that go wrong

• Linux or WSL reports something about run-detectors. The kernel handed your llamafile to the wrong interpreter, usually because WINE has registered a binfmt_misc handler for files beginning with MZ and llamafiles begin with MZ. The documented fix is to register the APE loader with the kernel explicitly; on WSL, the interop settings are the equivalent knob. Nothing is wrong with the download. The tell is that the error names an interpreter you never asked for.
• macOS refuses to run it. Gatekeeper treats it as downloaded software, so approve it in the security settings; the troubleshooting documentation also covers cases needing the Xcode command line tools, and a zsh-specific quirk when invoking it.
• Windows will not run a large one. Executables above 4 GB cannot run on Windows at all. Use the bare runtime with -m as above rather than looking for a bigger single file.