An Ollama Model Works via CLI but Not via the API

ollama run llama3.2 "hello" answers instantly. The same model through POST /api/generate returns an empty string, a 404, or a body your client cannot parse. The CLI is an HTTP client too — it is just filling in fields you are not.

First: are they the same server?

Before comparing request bodies, confirm that both are reaching the same process. The CLI honours OLLAMA_HOST, so if that is set in your shell — pointing at a remote box, a different port, or a container — the CLI is talking to one server and your code is talking to another with a different set of models pulled.

echo "$OLLAMA_HOST"
curl -s http://localhost:11434/api/tags | head -c 400
ollama list

If ollama list and /api/tags show different models, that is your answer and nothing else on this page applies. The same mechanism produces a connection refused from a container: Ollama binds to 127.0.0.1 by default, so a client outside the host namespace cannot reach it until OLLAMA_HOST=0.0.0.0 is set on the server. Binding wider than a trusted network is a real exposure — the API has no authentication — so bind to a specific interface rather than to everything if the machine is reachable.

The streaming default, and the empty result it causes

/api/generate and /api/chat stream by default. The response is newline-delimited JSON: one object per chunk, each with a fragment of the answer, ending with an object where done is true. A client that reads the body once and parses it as a single JSON document fails — either with a parse error about trailing data, or, if it is lenient, by returning only the first chunk, whose content is often an empty string.

That is the mechanism behind “the API returns nothing”. Set stream explicitly:

curl -s http://localhost:11434/api/generate -d '{
  "model": "llama3.2:latest",
  "prompt": "Name three prime numbers.",
  "stream": false
}' | jq -r '.response'

There is a second, unrelated cause of a fast empty response: the model was not resident and the request arrived while it was being loaded. Reproduce it by stopping the model first and sending a request immediately. Warm the model with a trivial request, or set keep_alive so it stays resident between calls, rather than treating the empty answer as a formatting problem.

The endpoint decides whether a template is applied

The CLI is a chat client. It builds a message list and lets the server apply the model’s chat template. Over HTTP you choose:

• /api/chat takes messages, applies the model’s template, and is the closest equivalent to ollama run. Use it for instruction-tuned models.
• /api/generate takes a single prompt and applies the model’s template around it — unless you disable that.
• raw: true on /api/generate means no formatting is applied and your prompt reaches the model exactly as written. Set it on an instruction-tuned model without writing the turn markers yourself and you get rambling, drift, or an immediate stop.
• template overrides the Modelfile’s template for that request. A half-correct override is worse than none, because it produces output that looks nearly right.

Sending messages to /api/generate is a common mistake with a confusing outcome: the field is not part of that endpoint’s schema, so the request has an empty prompt and the model answers a question you did not ask. Check that the field names match the endpoint before assuming the model is at fault. If the answer is empty rather than wrong, the empty-response page covers the model-side version of the same symptom.

The tag the CLI resolves and you do not

ollama run llama3.2 resolves to llama3.2:latest. The API does not do that resolution for arbitrary inputs in every version, and a name that does not match a pulled manifest returns a 404 whose body says the model was not found and suggests pulling it first.

Take the name from /api/tags rather than typing it. That endpoint returns exactly the strings the server will accept, including the tag. Two related traps: a model created from a Modelfile has whatever name you gave it and no :latest alias unless you made one; and a manifest directory that is present but damaged produces a model that is missing from listings while its blobs remain on disk, which is the manifest-not-found case.

If you are pointing an OpenAI-compatible client at Ollama’s /v1 endpoints, the model field still needs the Ollama name with its tag. A client configured with a hosted provider’s model string will 404 in a way that reads as a broken server.

Options, keep_alive and silently ignored fields

Generation parameters live in an options object, not at the top level. This is the highest-yield thing to check when the API answers but answers differently from the CLI:

curl -s http://localhost:11434/api/chat -d '{
  "model": "llama3.2:latest",
  "messages": [{"role": "user", "content": "Summarise this in one line."}],
  "stream": false,
  "keep_alive": "10m",
  "options": {
    "temperature": 0,
    "num_ctx": 8192,
    "num_predict": 256,
    "seed": 42
  }
}' | jq -r '.message.content'

A temperature placed at the top level is not an error — it is simply not read, so the model runs at its Modelfile default and you conclude the API behaves differently from the CLI. It does not; it received different parameters. num_ctx is the one that changes behaviour most visibly: a context set lower than your prompt silently truncates the beginning of it, which removes a system message and changes the answer in a way that looks like a different model.

Two more that bite. keep_alive controls how long the model stays resident; the CLI keeps a session alive while you type, so an API client that sends one request every few minutes pays a load on every call and can time out on the first. And format: "json" constrains decoding — useful, but a prompt that does not ask for JSON combined with that setting produces long runs of whitespace, which reads as a hang.