Running a Local Model From a Portable USB Drive

The requirement that makes this hard is not portability, it is “unmodified host”: no installer, no Python, no admin rights, no runtime left behind. One project is built for exactly that constraint, and one file-size limit decides how you lay the drive out.

Why llamafile and not a copied install

Copying an Ollama or llama.cpp installation to a stick fails on the host you most want it for. Ollama installs a background service. A llama.cpp build is compiled for one operating system and one architecture, so a Linux binary will not start on the Windows machine you have been handed. Anything Python-based needs an interpreter you are not allowed to install.

llamafile, a Mozilla project, exists to remove that class of problem. It combines llama.cpp with Cosmopolitan Libc to produce a single executable that runs unmodified on six operating systems — Windows, macOS, Linux, OpenBSD, FreeBSD and NetBSD — and on more than one CPU architecture, with no installation or configuration. In its simplest form the model weights are inside the executable, so one file is the entire deployment.

That simplest form is the one that will not work for you, which is the next section.

The 4 GB wall, twice

llamafile’s own documentation states the constraint plainly: only executables under 4 GB can run on Windows, so any llamafile above 4 GB will not work there. This is a property of the Windows executable loader, not of llamafile, and no version bump removes it.

Four gigabytes is below the useful line. A 7B or 8B model at Q4_K_M is around 4.9 GB — the published Q4_K_M file for Llama-3.1-8B is 4,920,739,232 bytes — so the single-file form caps you at roughly a 3B model at four bits, or a smaller model at a higher quant. That is a real option for classification and short-form work, and it is not an option for a general assistant.

The same number appears a second time for an unrelated reason. FAT32, which is what most USB sticks ship formatted as, cannot hold a file larger than 4 GiB at all. So a drive you did not reformat imposes the same ceiling on the weights even if you split them out of the executable. Two independent limits, the same number, and hitting either produces a different confusing failure.

The documented workaround for the first is to keep the binary and the weights separate: download the llamafile runtime on its own and point it at a GGUF with -m. The binary is small, the weights are a normal file, and neither is near 4 GB.

Choosing the filesystem first

This decision comes before anything else, because reformatting later means copying tens of gigabytes twice.

• FAT32 — readable everywhere without drivers, and capped at 4 GiB per file. Viable only for a small model. It also does not store a Unix execute bit, so on macOS and Linux you will be running chmod +x after every mount, or invoking through an interpreter.
• exFAT — the practical choice. No meaningful file size limit, and read-write support on current Windows, macOS and Linux. Same missing-execute-bit caveat as FAT32, which is a one-command annoyance rather than a blocker.
• NTFS or ext4 — full permissions, but each is awkward or read-only on the operating systems it is not native to, which defeats the point of a drive you hand to any machine.

Take exFAT unless every host is guaranteed Windows. Then plan the capacity from published file sizes rather than from the model name — budgeting disk for a model library has the arithmetic, and the short version is that a 64 GB stick holds about three usable models with room for the runtime.

Building the drive

1. Format the stick as exFAT and create two directories: bin/ for the runtime and models/ for the weights. Prefer USB 3.0 or better, and check which port you are in. USB 2.0 signals at 480 Mb/s, which is 60 MB/s before any protocol overhead, so a 4.92 GB model file cannot load faster than about 82 seconds on that bus and will take longer in practice. USB 3.0 raises the signalling rate tenfold and moves the bottleneck to the flash.
2. Put the llamafile runtime in bin/. Take the standalone runtime release rather than a model-bundled llamafile — that is the external-weights form and it is what keeps you under the Windows limit. Copy it as bin/llamafile.exe: Windows requires the .exe extension to execute it, and macOS and Linux do not care what it is called.
3. Put a GGUF in models/. Choose the level from what the machine you will plug into is likely to have, not from what your own machine has. Then verify its checksum — a USB transfer is precisely the path where a silent corruption is plausible, and this is the one moment you can still tell.
4. Start it. From the drive’s root, with the server bound to loopback only:

   # Windows
   bin\llamafile.exe -m models\Phi-3.5-mini-instruct-Q4_K_M.gguf \
     --server --host 127.0.0.1 --port 8080
   
   # macOS / Linux (the execute bit does not survive exFAT)
   chmod +x bin/llamafile
   ./bin/llamafile -m models/Phi-3.5-mini-instruct-Q4_K_M.gguf \
     --server --host 127.0.0.1 --port 8080

5. Confirm it answers before trusting the setup on a machine you do not control:

   curl -s http://127.0.0.1:8080/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{"model":"local","messages":[{"role":"user","content":"Reply with the word ready."}],"max_tokens":8}'

6. Write a one-line launcher for each platform — run.cmd and run.sh at the drive root — so the next person who plugs it in does not need to remember the flags. Include the chmod in run.sh, because it will be needed every time on a FAT-family volume.

What it will feel like

Two honest caveats, both about the drive rather than the model.

First, loading is slow and it is slow every time. llama.cpp memory-maps the weights file by default, which means pages are faulted in from the USB device as they are touched rather than read in one sequential stream. On internal storage that is a win; on a slow removable device it turns the first generation into a long pause with random-access reads. If the host has RAM to spare, forcing a full load up front behaves better — recent llama.cpp builds fold the old --mlock and --no-mmap flags into a single -lm, --load-mode, so check --help on your build for the spelling it accepts.

Second, the host’s hardware decides the speed and you will not know it in advance. The ceiling is that machine’s memory bandwidth divided by your model’s file size, as derived in what to expect from CPU-only inference, and on an arbitrary office laptop that is a modest number. Plan for CPU-only: the point of this drive is that it runs anywhere, and anywhere includes machines with no usable GPU.