Running a Local Model as a systemd Service That Survives Reboot

Running a model server from a terminal works until you close the terminal, and works until the machine reboots. A systemd unit fixes both, but the default restart and timeout settings are tuned for programs that start in a second, and a model server is not one.

The unit file

1. Create an unprivileged account for the service to run as.

   sudo useradd --system --no-create-home --shell /usr/sbin/nologin llama
   sudo install -d -o llama -g llama /srv/models

2. Write the unit. Every path is absolute, because systemd does not run a shell: ~ is not expanded, $PATH is not your login PATH, and a bare command name may not resolve.

   # /etc/systemd/system/llama-server.service
   [Unit]
   Description=llama.cpp model server
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   Type=exec
   User=llama
   Group=llama
   EnvironmentFile=/etc/llama-server.env
   ExecStart=/usr/local/bin/llama-server \
     --model /srv/models/qwen2.5-7b-instruct-q4_k_m.gguf \
     --host 127.0.0.1 --port 8080 \
     --ctx-size 8192 --parallel 2 \
     --api-key-file /etc/llama-server.keys \
     --no-webui
   
   Restart=always
   RestartSec=5
   StartLimitIntervalSec=0
   TimeoutStartSec=600
   TimeoutStopSec=60
   LimitMEMLOCK=infinity
   
   [Install]
   WantedBy=multi-user.target

3. Load it, enable it for boot, and start it in one command.

   sudo systemctl daemon-reload
   sudo systemctl enable --now llama-server
   systemctl status llama-server
   journalctl -u llama-server -f

Wants= and After= both name network-online.target on purpose. After= alone only orders the unit relative to a target that may never be pulled into the transaction; Wants= is what causes it to be. This matters less when the server binds to loopback and a great deal when it binds to a specific address that does not exist yet at the moment it starts.

Restart semantics and the rate limit

Restart=on-failure is the instinct and it is the wrong choice here. It restarts only on a non-zero exit or a signal, so a server that decides to shut down cleanly — after an unrecoverable runtime error it handled, or because a supervisor sent it a termination it treated as normal — exits 0 and stays down. Restart=always restarts regardless of how it exited, which is what “this must be running” actually means.

The setting that surprises people is the rate limit sitting behind it. systemd will refuse to keep restarting a unit that fails repeatedly: by default, more than DefaultStartLimitBurst starts (5) within DefaultStartLimitIntervalSec (10 seconds) puts the unit into a failed state and leaves it there, with the log line start request repeated too quickly. A model server that fails in under two seconds — a missing file, a bad flag, not enough VRAM — hits that limit immediately, and the outcome is a service that is permanently down after a transient problem.

Two ways to handle it, and they are not equivalent:

• StartLimitIntervalSec=0 disables the limiter entirely. Combined with a RestartSec of a few seconds, the unit retries forever. Right for a server whose failure is usually transient, such as one that races the GPU driver at boot. It also means a permanently broken configuration retries silently forever, so the journal is where you find out.
• A longer RestartSec keeps the limiter but spaces the attempts so five of them cannot fit inside the window. RestartSec=30 with the defaults gives you unlimited retries in practice while still failing loudly if something restarts in a tight loop for a different reason.

Note also that StartLimitIntervalSec belongs in the [Unit] section on older systemd versions and is accepted in [Service] on newer ones. If it appears to be ignored, that is the first thing to check; systemd-analyze verify /etc/systemd/system/llama-server.service reports directives it did not understand.

The start timeout that kills a cold load

The second default that bites is TimeoutStartSec, which is 90 seconds unless you say otherwise. A large GGUF being read from a cold page cache on a slow disk can take longer than that to become ready, and when the timeout expires systemd kills the process it is waiting for — mid-load, every time, always on the first boot and never when you test it warm.

Type=exec considers the service started once the binary has been executed successfully, which is a good fit for a server that has no readiness protocol; it does not wait for the port to open. If you want systemd to know when the model is genuinely ready, the honest options are a generous TimeoutStartSec as above, or a separate health-check unit that curls the endpoint. Why the first load is so much slower than the second is covered in model loading and warmup, and cold-start latency has the numbers to plan around.

TimeoutStopSec is the mirror image and matters for the reason set out in GPU memory not freed after a crash: if systemd escalates to SIGKILL before the server has released its CUDA context, you get the stuck-allocation case on every restart.

Environment, secrets and paths

Anything set with Environment= is visible to any user who can run systemctl show llama-server. An API key does not belong there. Use EnvironmentFile= pointing at a file the service account can read and nobody else can:

sudo tee /etc/llama-server.env >/dev/null <<'EOF'
LLAMA_ARG_THREADS=8
CUDA_VISIBLE_DEVICES=0
EOF
sudo chown root:llama /etc/llama-server.env
sudo chmod 640 /etc/llama-server.env

LimitMEMLOCK=infinity is only needed if you pass --mlock to keep the model resident and unswappable; without the raised limit the lock fails and the flag silently achieves less than you intended.

Two directives are worth adding once you are past the first working version. StateDirectory=llama makes systemd create and own /var/lib/llama with the right permissions before the service starts, which is a better place for a prompt cache or a slot save directory than a path you created by hand and will forget to recreate on a new machine. And if you run more than one model, use a template unit — a file named [email protected] whose ExecStart reads /etc/llama/%i.conf — so that systemctl enable --now llama-server@coder and llama-server@general are two instances of one definition rather than two files that will drift apart.

Resist the temptation to reach for Type=notify unless the program actually implements the readiness protocol. systemd will wait for a notification that never arrives, hit TimeoutStartSec, and kill a server that was working perfectly — a failure that looks like the model being slow to load and is not. The same caution applies to WatchdogSec: it is only useful against a program that pings it.

For Ollama, do not edit the unit the installer wrote — it will be replaced on the next upgrade. Use a drop-in, which is what systemctl edit creates:

sudo systemctl edit ollama.service
# in the editor, add only:
[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl daemon-reload
sudo systemctl restart ollama
systemctl cat ollama.service   # shows the shipped unit plus your drop-in

Verify both failure modes

A unit that has not been tested against the two events it exists for is a unit you are guessing about. Both tests take a minute.

1. Crash. Kill the process the way a crash would, and confirm systemd brings it back rather than that you brought it back.

   systemctl show -p MainPID --value llama-server
   sudo systemctl kill -s SIGKILL llama-server
   sleep 15
   systemctl show -p MainPID --value llama-server   # a different PID
   systemctl show -p NRestarts --value llama-server

2. Reboot. The only test for WantedBy=multi-user.target having taken effect is a real restart. Confirm it was enabled first, so a failure tells you something.

   systemctl is-enabled llama-server   # enabled
   sudo reboot
   # after it comes back
   systemctl is-active llama-server
   journalctl -u llama-server -b --no-pager | head -40

3. Break it on purpose. Point --model at a path that does not exist, reload, and watch what the journal says. This is how you find out whether your restart settings produce a useful loop or a silent failed unit, and it is much better to learn that now.