Jan, Start to First Response

Jan is a desktop chat application with llama.cpp bundled inside it. One installer gets you an inference engine, a model browser and an OpenAI-compatible server, and none of the three needs an account to work.

Install, and what comes bundled

Builds exist for macOS on Apple Silicon and Intel, Windows and Linux. The important structural fact is that the installer carries the inference engine with it: Jan’s documentation describes llama.cpp as its local engine, so there is no separate backend to install, no Python, and no CUDA toolkit to match against a driver. That is the whole design difference between Jan and a front end like Open WebUI, which assumes you already have a server running somewhere.

The consequence is that the engine is versioned with the app. When a model needs a newer llama.cpp than the one shipped, the fix is updating Jan rather than updating a runtime underneath it. Jan exposes the engine settings rather than hiding them, so the thread count, the GPU layer count and the context size are all reachable from the UI rather than only from a command line.

Downloading a model from the Hub

1. Open the Hub tab in the left sidebar. It lists models with their sizes; downloaded ones then appear automatically in the model picker.
2. Choose by memory, not by name. The same budget applies here as everywhere in this cluster: weights plus KV cache have to be resident, and the cache grows linearly with the context length you set. The full derivation is on the combined memory budget page and the quantisation trade-off is in choosing a quantisation.
3. Wait for the download to finish before selecting it. A partially downloaded GGUF fails at load with a format error rather than a length error, which sends people looking in the wrong place.
4. If the model you want is not in the Hub, Jan can also take a GGUF from Hugging Face or from disk. That is the escape hatch worth knowing about early, because a curated hub always lags the ecosystem by a few weeks and the thing you read about this morning will not be in it yet.

Size selection is where most first attempts go wrong, and the failure is not obvious: a model that is slightly too large does not refuse to load, it loads with part of itself on the CPU and runs at a fraction of the speed. If a download that should be fast is producing a token every second or two, suspect the split before you suspect the model.

First response

Start a new thread, pick the model, send a message. The model loads on the first message of a thread, so the first response is slower than every subsequent one; that pause is the weights being read, not the model thinking.

Before you judge quality, check the context length in the model’s settings. Desktop applications generally load at a modest default rather than at the model’s trained maximum, because the maximum would not fit on most machines. A model that seems to forget the start of a conversation after a few turns is a model whose window is smaller than the conversation, which is a setting rather than a defect — the same failure LM Studio reports explicitly and discusses on the context overflow page.

The local API server on 1337

Jan can serve the loaded model to other applications. Its Local API Server documentation gives the procedure and the defaults: open Settings, then Local API Server, choose a model and press Start Server. The log line JAN API listening at http://127.0.0.1:1337 is the confirmation. The API prefix is /v1, following the OpenAI convention.

curl http://127.0.0.1:1337/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3:8b",
    "messages": [{"role": "user", "content": "say hello in six words"}],
    "stream": false
  }'

Two documented defaults are worth knowing. The API key field may be left empty, which disables authentication — fine on loopback, not fine the moment the server is reachable from elsewhere. And CORS is enabled by default, which is what allows a page in a browser to call the server directly; if you are not building a browser client, there is no reason to leave it on.

Because the surface is OpenAI-shaped, an existing client usually needs only a base URL and a dummy key to point at it. What it will not inherit is the provider-specific extras: fields that exist on a hosted API and not in llama.cpp’s server are ignored rather than honoured, and a request that depends on one will succeed while quietly doing something else. Check the response, not just the status code, the first time you repoint a client.

Settings that decide how local it stays

• Verbose logging is on by default and records requests and responses. That is genuinely useful when a client will not connect and is exactly the wrong default if the conversations are sensitive, because it writes them to disk in plain text. Decide rather than inherit.
• Remote providers. Jan can also talk to hosted APIs with your own keys. That is a feature, not a leak, but it means “Jan is local” is true of a configuration rather than of the application. If the guarantee matters, check which provider a thread is actually using before you type into it.
• Context length and GPU layers live in the model’s settings rather than in a global preference, so they are per model. Raising one model’s window does not raise another’s, and the memory competition between them is real if you keep several loaded.