Running a Local LLM on a NAS

A NAS is an always-on x86 machine with a lot of storage, modest RAM and usually no GPU. That is enough to serve a small quantized model to your house, and not enough for anything else. The arithmetic below tells you which of those you are about to get.

Find out whether your box can do this at all

Three facts decide it. Get them from the machine rather than from a spec sheet, over SSH:

# architecture and instruction set — ARM units cannot run
# most prebuilt inference binaries, and AVX2 matters a lot on x86
lscpu | grep -E 'Architecture|Model name|avx2'

# how much RAM is actually free after the NAS's own services
free -g

# is there any GPU at all?
lspci | grep -Ei 'vga|3d|display'

• ARM is a hard stop for the common prebuilt images. Some entry-level and many rack units are ARM. Check before anything else.
• AVX2 changes the answer by a large factor. llama.cpp’s CPU kernels use wide SIMD; a chip without AVX2 falls back to much slower paths. An older Celeron-based NAS will technically work and will not be pleasant.
• Free RAM, not total RAM. The NAS operating system, its indexing, its container runtime and any other services are already resident. Budget from what free -g shows available, and leave the box enough to keep doing its actual job — a NAS that has been OOM-killed into unresponsiveness while serving a model has failed at the thing you bought it for.

Derive the ceiling before you build

Generating one token requires reading every active weight from memory once, so throughput at batch size one is bounded by memory bandwidth divided by model size. NAS memory configurations are modest and usually single- or dual-channel DDR4, which makes the ceiling easy to compute and sobering to look at. The bits-per-weight figure below is the measured one, for the reasons in why file size never matches parameter count.

Assumptions, all of which you should replace with your own:
  dual-channel DDR4-2666
  = 2 channels x 8 bytes x 2666 MT/s
  = 42.7 GB/s theoretical peak

Model: a 3B at Q4_K_M
  = 3e9 x 4.8944 bits / 8       (measured bits per weight,
  = 1.84 GB of weights           not the 4 in the name)

Upper bound on generation:
  42.7 / 1.84 = 23 tokens/s

Same box, an 8B at Q4_K_M (4.9 GB):
  42.7 / 4.9  = 8.7 tokens/s

These are ceilings assuming perfect bandwidth utilisation and ignoring the KV cache, attention arithmetic and every other cost. Real throughput is meaningfully below them, and a single-channel configuration halves them again. Nobody publishes measured tokens per second for your particular NAS and model combination, so measure it yourself once it is running: llama-bench -m model.gguf -p 128 -n 128 reports prompt processing and generation separately, which is the distinction that matters when a long prompt feels slow but the reply streams fine.

The practical conclusion: aim at 1B–4B models at Q4, treat 7B/8B as the outer edge, and do not attempt anything larger. If the ceiling above is not enough for what you want, the answer is different hardware, covered in the home lab tiers.

Synology: Container Manager

Synology’s Docker package was replaced by Container Manager in DSM 7.2, and it accepts a Compose project directly, which is easier to reproduce than clicking through the GUI.

1. Install Container Manager from Package Center. Create a shared folder for the models — on an SSD volume or an SSD cache if you have one, because the weights are read in full on every cold start.
2. Create docker-compose.yml in that folder:

   services:
     ollama:
       image: ollama/ollama:latest
       container_name: ollama
       restart: unless-stopped
       ports:
         - "11434:11434"
       volumes:
         - /volume1/docker/ollama:/root/.ollama
       environment:
         - OLLAMA_KEEP_ALIVE=-1
         - OLLAMA_MAX_LOADED_MODELS=1
       deploy:
         resources:
           limits:
             memory: 6g

   OLLAMA_KEEP_ALIVE=-1 keeps the model resident instead of unloading it after an idle period, which on a NAS matters more than elsewhere: re-reading two gigabytes off the array on every request after a gap is the difference between a usable assistant and one that takes half a minute to wake up. The memory limit is there so that a runaway container is the container’s problem rather than the file server’s.
3. In Container Manager, create a project pointing at that folder and start it.
4. Pull a small model and confirm it answers — the model naming and Modelfile conventions are in the Ollama guide:

   sudo docker exec -it ollama ollama pull qwen3:4b
   
   curl http://NAS-IP:11434/api/generate -d '{
     "model": "qwen3:4b",
     "prompt": "Reply with one word: ready",
     "stream": false
   }'

5. Then close it down. The port above has no authentication of any kind. Either restrict it to your LAN at the firewall, or put it behind DSM’s reverse proxy with authentication in front. Do not expose it to the internet.

Unraid: Docker and an optional GPU

Unraid runs ordinary Docker and gives you something Synology mostly does not: a real PCIe slot in a real case, so a GPU is on the table. That changes the tier entirely — a NAS with a 12 GB card is a tier 1 machine that happens to also hold your files.

1. Install the Nvidia Driver plugin from Community Applications and reboot, then confirm the driver sees the card with nvidia-smi from the Unraid terminal.
2. Note the GPU’s UUID from nvidia-smi -L. Unraid’s convention is to pass that UUID rather than a device index, because indices move between boots.
3. Add the container with the runtime and the device:

   docker run -d --name ollama --restart unless-stopped \
     --runtime=nvidia \
     -e NVIDIA_VISIBLE_DEVICES=GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx \
     -e OLLAMA_KEEP_ALIVE=-1 \
     -v /mnt/user/appdata/ollama:/root/.ollama \
     -p 127.0.0.1:11434:11434 \
     ollama/ollama:latest

   Binding the published port to 127.0.0.1 keeps it off the LAN until you have decided how it should be reached.
4. If a virtual machine on the same box already has the card passed through, this will fail. One consumer GPU cannot be bound to a VM and shared with Docker at the same time; pick one.

What goes wrong on a NAS specifically

• Models on the spinning array. The default appdata location may live on rotating disks. Cold start then means reading gigabytes at platter speed, and the array may have spun down entirely. Put weights on SSD.
• Compression and deduplication. Quantized weights are effectively incompressible. Filesystem compression on that dataset spends CPU to save nothing, and deduplication spends memory you do not have.
• Silent OOM. Without a memory limit, the container can be killed by the kernel mid-request, restart, and look fine. The symptom is intermittent failures with no error in the container log. Check dmesg for the OOM killer.
• Backup sweeping up the models. Model weights inside a backed-up share means uploading tens of gigabytes of files you can re-download. Exclude the model directory explicitly.
• Thermals. NAS chassis are designed for the thermal load of disks, not of a CPU pinned at 100% for minutes. Sustained inference will throttle in a small enclosure, which shows up as throughput that gets worse the longer you use it.