Securing a Local Model Server Exposed to Your LAN

Local model servers are built for the machine they run on. Their defaults assume the only caller is you, on loopback, and every one of those assumptions breaks in the same edit — the one where you change the bind address so your laptop can reach the desktop.

What binding to 0.0.0.0 actually exposes

llama.cpp’s llama-server documents --host as defaulting to 127.0.0.1 and --port to 8080, and --api-key as “default: none”. Ollama listens on 127.0.0.1:11434 unless OLLAMA_HOST says otherwise. Both of those defaults are a security control, and it is the only one either program ships with.

When you change the bind address, what becomes reachable is not just “the model”. On llama.cpp it is the completion endpoints, the bundled web UI, and /props, which reports the loaded model and the server’s configured chat template — useful reconnaissance for anyone deciding whether the box is worth more attention. On Ollama it is worse in kind rather than in degree, because the HTTP API is a write API: the same unauthenticated interface that serves /api/generate also serves /api/pull and /api/delete. Anyone on the network segment can make your machine download arbitrary model weights until the disk is full, or remove the ones you have.

“It is only my LAN” is doing a lot of work in that sentence. A LAN contains guest devices, a smart TV, whatever a housemate installed, and—if the router has UPnP enabled and something asked for a mapping—possibly the open internet. Treat the boundary as untrusted and the rest of this page is short.

Step one: turn the key on

llama.cpp accepts --api-key with a comma-separated list, or --api-key-file pointing at a file with one key per line and # comments. With either set, requests must carry a bearer token. Generate a key that is not a word:

1. Create the key and a file only the service account can read.

   openssl rand -hex 32 | sudo tee /etc/llama-server.keys >/dev/null
   sudo chown llama:llama /etc/llama-server.keys
   sudo chmod 600 /etc/llama-server.keys

2. Start the server bound to loopback, with the key file. Loopback is deliberate: the proxy in step two is what listens on the network.

   llama-server \
     --model /srv/models/qwen2.5-7b-instruct-q4_k_m.gguf \
     --host 127.0.0.1 --port 8080 \
     --api-key-file /etc/llama-server.keys \
     --no-webui \
     --ctx-size 8192 --parallel 2

3. Confirm that an unauthenticated request is refused and an authenticated one is not.

   curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v1/models
   # 401
   
   curl -s http://127.0.0.1:8080/v1/models \
     -H "Authorization: Bearer $(sudo head -n1 /etc/llama-server.keys)"

--no-webui is not decoration. The web UI is a convenience for a machine you are sitting at; on a networked server it is another surface and another thing to reason about. Ollama has no equivalent key flag, which is precisely why the proxy below is not optional for it — the authentication has to live somewhere, and if the server cannot do it, the thing in front of it must.

Step two: TLS in front of it

A bearer token sent over plain HTTP is a bearer token readable by anyone on the path, and on a wireless LAN “the path” is generous. Terminate TLS. llama.cpp can do it itself with --ssl-key-file and --ssl-cert-file, but only if the binary was built with OpenSSL support enabled (the project documents this as the -DLLAMA_OPENSSL=ON CMake flag), and a prebuilt release generally was not. A separate terminator is the more portable answer and gives you somewhere to put the rest of the policy.

# /etc/caddy/Caddyfile
llm.home.arpa {
    tls internal
    reverse_proxy 127.0.0.1:8080
}

tls internal tells Caddy to issue the certificate from its own local certificate authority rather than from a public one, which is the right choice for a name that does not resolve on the public internet. Every client that will call the server has to trust that CA root once; Caddy writes it into its data directory and documents where to find it. Skipping that step and passing curl -k everywhere converts TLS into decoration, because a client that accepts any certificate accepts the attacker’s. The mechanics of issuance, streaming and timeouts are the subject of the reverse-proxy page; this page only needs the terminator to exist.

Step three: close the port to everything else

Two things are now listening: the proxy on 443, and the model server on 8080. The second is bound to loopback, so a host firewall is belt-and-braces rather than the primary control — but it is the control that survives somebody later editing the systemd unit and reintroducing --host 0.0.0.0 without thinking about it.

1. Deny by default, allow the proxy, allow your own subnet only.

   sudo ufw default deny incoming
   sudo ufw allow from 192.168.1.0/24 to any port 443 proto tcp
   sudo ufw deny 8080/tcp
   sudo ufw enable
   sudo ufw status numbered

2. Verify from another machine that the model port is unreachable and the proxy is not.

   nc -zv 192.168.1.50 8080   # refused or filtered
   curl -sS https://llm.home.arpa/v1/models -H "Authorization: Bearer $KEY"

3. Check what is actually listening, which is the only answer that does not depend on a config file being what you think it is.

   sudo ss -ltnp | grep -E 'llama|caddy|ollama'

If the server is Ollama, add OLLAMA_HOST=127.0.0.1:11434 to the unit explicitly rather than relying on the default, and point the proxy at that. The reason to be explicit is that the default has changed shape across releases and an environment variable set by an installer script is easier to audit than an absent one. Setting environment for a service without editing the shipped unit file is covered on the systemd page.

What this does not protect against

• A single shared key is a single shared identity. There is no per-user attribution, no revocation of one caller, and no rate limit. If four devices hold the same key, rotating it because one was lost breaks all four.
• Prompt injection is unaffected. Authentication decides who may call the model. It says nothing about what the text they send instructs the model to do, and a local model with tool access is as susceptible as a hosted one.
• The weights are still files. Anyone with shell access, or any process running as your user, can read the GGUF directly. Where a model is licensed for your use rather than redistribution, the filesystem permissions are the control, not the API key.
• Denial of service is trivially available. One authenticated caller can occupy every server slot with long generations. Slot exhaustion looks exactly like the server hanging; how slots are allocated is worth understanding before you pick a --parallel value.
• Nothing here is logged usefully by default. If you need to know later which device made which request, that has to be added at the proxy, because the model server does not record an identity it was never given.