Running a Local Model Behind a Reverse Proxy for Remote Access

Putting a proxy in front of a model server is a five-line configuration that works immediately and then fails in three specific ways, all of them caused by LLM traffic being unlike the web traffic the defaults were written for: responses that stream for minutes, requests that are large, and connections that must stay open with long gaps between bytes.

What the proxy has to get right

The architecture is fixed and simple. The model server binds to loopback and speaks plain HTTP; the proxy listens on the network, terminates TLS, and forwards. That split means the model server is unreachable except through the proxy, which is what makes the authentication story on securing a local model server enforceable rather than advisory.

Four requirements are peculiar to this traffic and none of them are defaults:

• Responses must not be buffered. A streamed completion is useful precisely because tokens arrive as they are produced. A proxy that accumulates the response and forwards it in one piece produces correct output with all of the perceived latency benefit destroyed.
• Read timeouts must exceed a long generation. A reasoning model thinking for two minutes before its first token looks exactly like a dead upstream to a proxy with a 60-second default.
• Request bodies can be large. A long context, a pasted file or a base64 image easily exceeds the default body limit some proxies impose, and the failure is a flat rejection before the model is ever consulted.
• TLS has to be issued for a name that may not be public. The challenge type available to you depends on whether the machine is reachable from the internet, which is usually the whole reason you are reading this.

The Caddy configuration

1. Confirm the model server is on loopback and answering, so that a later failure is unambiguously the proxy’s.

   curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/health

2. Write the Caddyfile. For a name that resolves publicly and a machine reachable on port 80 or 443, this is the whole configuration — Caddy obtains and renews the certificate on its own.

   # /etc/caddy/Caddyfile
   llm.example.com {
       reverse_proxy 127.0.0.1:8080
   
       request_body {
           max_size 32MB
       }
   }

3. For a machine behind NAT with no inbound port 80, use the DNS challenge instead, which proves control of the name through your DNS provider rather than through an inbound connection. This requires a Caddy build including your provider’s DNS module. For a purely internal name, replace the whole block with tls internal and distribute Caddy’s local CA root to your clients.

   llm.example.com {
       tls {
           dns cloudflare {env.CF_API_TOKEN}
       }
       reverse_proxy 127.0.0.1:8080
   }

4. Validate, then reload without dropping connections.

   caddy validate --config /etc/caddy/Caddyfile
   sudo systemctl reload caddy
   journalctl -u caddy -n 50 --no-pager

Caddy’s documentation records two behaviours that matter here and that you get without configuring them. It sets X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host by default, and it ignores the values a client supplied for them, to prevent spoofing. And although no periodic flushing is done by default, responses are flushed immediately when the response carries Content-Type: text/event-stream, when the Content-Length is unknown, or when both sides speak HTTP/2 with an unknown length and no compression. Streamed completions satisfy the first two, which is why streaming through Caddy works with no buffering directive at all.

The nginx equivalent

nginx does not make the same assumption, and the configuration below is longer for exactly that reason.

server {
    listen 443 ssl;
    http2 on;
    server_name llm.example.com;

    ssl_certificate     /etc/letsencrypt/live/llm.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;

    # a long prompt or an image is not 1 MB
    client_max_body_size 32m;

    location / {
        proxy_pass http://127.0.0.1:8080;

        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # the three that matter for streaming
        proxy_buffering    off;
        proxy_cache        off;
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
    }
}

Certificates here come from a separate ACME client such as certbot, which is the structural difference from Caddy: issuance and renewal are somebody else’s job, and a renewal that fails silently is a failure mode Caddy does not have.

Streaming, and the failure everybody hits

proxy_buffering is on by default in nginx. With it on, nginx reads the upstream response into its own buffers and forwards it when a buffer fills or the response ends. For a normal web page that is a performance feature. For a server-sent-event stream of token deltas, it means the client receives nothing for the entire generation and then receives everything at once. The request succeeds, the output is correct, and streaming is silently not happening — which is why this is diagnosed late, often after somebody concludes the model got slower.

proxy_buffering off; is the fix. An upstream can also request it per response by sending the X-Accel-Buffering: no header, which nginx honours; that is useful when the same proxy serves both streaming and non-streaming routes and you would rather not disable buffering globally.

The timeout is the second half of the same problem. proxy_read_timeout defaults to 60 seconds and is measured between successive reads from the upstream, not for the whole response. A streaming completion emitting a token every 50 ms never approaches it. A reasoning model that thinks for ninety seconds before emitting anything exceeds it and the connection is closed mid-request, with a 504 for the client and an upstream timeout in the error log. Raise it well past your worst case.

proxy_http_version 1.1 with an empty Connection header is the third piece: nginx proxies with HTTP/1.0 by default, which does not support chunked transfer encoding upstream and interacts badly with keepalive. Getting all three wrong produces a proxy that appears to work in every test that does not involve streaming.

Verify from outside

1. Check the certificate chain and the negotiated protocol from a machine that is not the server.

   curl -sSv https://llm.example.com/health 2>&1 | grep -E 'subject|issuer|ALPN|HTTP/'

2. Prove the stream is a stream. --no-buffer disables curl’s own buffering, so what you see is what arrived when it arrived; the output should appear progressively rather than in one burst at the end.

   curl -N --no-buffer https://llm.example.com/v1/chat/completions \
     -H "Authorization: Bearer $KEY" \
     -H 'Content-Type: application/json' \
     -d '{"model":"local","stream":true,
          "messages":[{"role":"user","content":"Count slowly to twenty."}]}'

3. Prove the body limit is real by sending something large, and check you get a model response rather than a 413.

   python3 -c "import json,sys; json.dump({'model':'local','messages':[{'role':'user','content':'x'*200000}]}, sys.stdout)" \
     > /tmp/big.json
   curl -sS -o /dev/null -w '%{http_code}\n' https://llm.example.com/v1/chat/completions \
     -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
     --data-binary @/tmp/big.json

4. Prove the long-generation case survives the timeout by requesting a long answer and timing it, rather than waiting to discover it in use.