Running llama.cpp's Server in OpenAI-Compatible Mode

llama-server speaks enough of the OpenAI HTTP shape that most client libraries work against it unchanged by pointing base_url at your own machine. The gaps are specific and worth knowing before you find them at runtime.

Starting the server

The minimum is a model and nothing else, because the defaults now do a lot of work: -ngl defaults to auto, -c defaults to the model’s trained context, --host is 127.0.0.1, --port is 8080, and --jinja — which makes the server use the chat template embedded in the GGUF — is enabled.

llama-server -m ./models/qwen2.5-7b-instruct-q4_k_m.gguf \
  -c 8192 \
  -ngl 99 \
  --host 127.0.0.1 --port 8080 \
  --api-key "$LLAMA_API_KEY"

Every one of those flags also has an environment variable, which is what you want in a container: LLAMA_ARG_CTX_SIZE, LLAMA_ARG_N_GPU_LAYERS, LLAMA_ARG_PORT and so on. Wait for the line saying the server is listening before you call it — the HTTP port opens after the model is loaded and the KV cache allocated, which on a large model is not instant.

The call, and what comes back

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LLAMA_API_KEY" \
  -d '{
    "model": "local",
    "messages": [
      {"role": "system", "content": "Answer in one sentence."},
      {"role": "user", "content": "Why is prefill faster than generation?"}
    ],
    "temperature": 0.2,
    "max_tokens": 200
  }'

The response is the familiar envelope: an id, a choices array whose first element has a message and a finish_reason, and a usage object with prompt_tokens, completion_tokens and total_tokens. The model field in the request is ignored for routing — one server process serves one model, and whatever string you send is echoed back. Clients that validate model names against a list should call GET /v1/models, which the server implements.

Add "stream": true and you get server-sent events in the OpenAI delta shape, terminated by the literal data: [DONE] line. Alongside the OpenAI surface the server also exposes /v1/completions, /v1/embeddings and, on current builds, /v1/responses — plus its own native endpoints, which expose llama.cpp-specific parameters the OpenAI schema has no field for.

What OpenAI-compatible does and does not mean

Compatible means the request and response schemas match closely enough for a client library to parse them. It does not mean behavioural parity, and the differences are systematic rather than random:

• Sampling parameters are a superset in one direction and a subset in the other. llama.cpp supports parameters OpenAI has no name for — min_p, repeat_penalty, mirostat, tfs_z — which you pass as extra fields and most clients allow. Conversely, some hosted-only fields are accepted and ignored.
• Structured output is a grammar underneath. response_format with a json_schema is converted to GBNF and enforced by the sampler, so the guarantee is stronger than a prompt instruction but the schema features supported are those the converter handles. See GBNF grammars.
• Tool calling depends on the model’s template. The server can parse tool calls out of a model’s native format into the OpenAI tool_calls shape, but only for the formats it knows. A model whose template emits tool calls in an unusual form will return them as ordinary message text.
• There are no organisation-level concepts. No projects, no per-key rate limits, no usage dashboard. One key, one model, one process.

Chat templates are where compatibility breaks

The single most common cause of “it works but the answers are bad” is a wrong chat template. The server takes your messages array and renders it into the one flat string the model actually sees, using the Jinja template stored in the GGUF. Mismatched special tokens do not error — the model just receives a prompt in a format it was not trained on and behaves like a base model having a bad day.

--chat-template overrides the embedded template by name, and --chat-template-file takes one from disk. Reach for these only when the GGUF was converted without a template or with the wrong one; the embedded template is right far more often than a manual guess. If system messages seem to be ignored, that is the specific symptom to check first — some templates place the system message somewhere the model weights it differently, and a few older ones drop it.

Concurrency, slots and the context you actually get

The server splits its KV cache into slots and serves one sequence per slot. -np/--parallel sets the count and defaults to auto. This is the arithmetic people miss: with -c 32768 -np 4, each slot gets 8192 tokens, not 32768. A request longer than a slot’s share fails or is truncated even though the total looks generous.

Slots also carry prompt-prefix reuse across requests, which is where a shared system prompt stops being recomputed every call — the same economics as prompt caching on hosted providers, with the cache in your own RAM. The trade is that a slot holding a long cached prefix is not available to a request that needs a different one; see server slots.

Do not put this on the internet by accident

--host defaults to 127.0.0.1, and the usual first move when a container cannot reach it is to change that to 0.0.0.0. That binds every interface. --api-key defaults to none, so at that point anyone who can route to the port has unmetered use of your GPU and can read anything in the prompt cache. Bind to a private interface, set a key, and put a reverse proxy in front if it needs to leave the machine at all.