LocalAI, Start to First Request

LocalAI is not a chat application. It is a server that implements OpenAI’s API surface over local backends, so that software written against OpenAI keeps working when you change nothing but the base URL. The end state of this page is a curl request answered by a process you own.

What LocalAI actually is

The distinction from the desktop runtimes in this cluster is structural. Jan and GPT4All are applications with a server bolted on; LocalAI is a server with no application, designed to be run as a container or a systemd unit and talked to over HTTP. It also spans more than text: the project ships backends for transcription, image generation and speech alongside chat, all behind the same API shape. The v4.8.2 release, dated 7 August 2026 on mudler/LocalAI’s releases page, is the current one at the time of writing and publishes binaries for Linux on AMD64 and ARM64 and macOS on ARM64.

That shape has a cost worth naming before you start: because it wraps several backends, LocalAI has more moving parts than a single binary that only runs GGUFs. When something fails it is usually the backend rather than the server, and the log line that matters names the backend rather than the endpoint. If you want a browser interface on top of it, that is a separate component: Open WebUI is the usual pairing, and it connects to LocalAI exactly as it connects to anything else OpenAI-shaped.

Getting it running

The documented container form binds the API to port 8080 and gives the container a name you can address later:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest

Two variations you will want soon. Mount a host directory at /build/models so downloaded weights survive the container being recreated — otherwise every restart is a fresh multi-gigabyte download. And if you have an NVIDIA GPU, use the CUDA-tagged image and pass --gpus all; the default image is CPU-only, which works and is slow, and people frequently conclude the project is slow when they have simply run the CPU image on a machine with a GPU in it.

docker run -p 8080:8080 --gpus all \
  -v "$PWD/models:/build/models" \
  --name local-ai -ti localai/localai:latest-gpu-nvidia-cuda-12

There is also a native binary, which is the better choice on a machine where you do not want Docker in the path between a request and a GPU. Either way the API lands on 8080 unless you say otherwise.

Installing a model

LocalAI has a gallery, and the CLI drives it. From the project’s getting started documentation:

local-ai models list                 # what the gallery offers
local-ai models install qwen3-4b     # fetch and configure one
local-ai run qwen3-4b                # install if needed, then serve

The gallery is not the only source. The same run command accepts URIs that point elsewhere, which is the part worth remembering because it removes the usual “my model is not in the list” dead end:

local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf
local-ai run ollama://gemma:2b
local-ai run oci://localai/phi-2:latest

The ollama:// scheme is the useful one if you already have an Ollama setup and do not want to download the same weights twice. Note that weights carry licences: some are gated and require you to accept terms on the source before the download will succeed, and the correct response to a 401 from Hugging Face is to go and accept the licence, not to look for a mirror.

The first request

1. Confirm the server sees the model. curl http://localhost:8080/v1/models returns the list in OpenAI’s shape. If your model is not there, nothing below will work and the problem is the install step.
2. Send a chat completion:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-4b",
    "messages": [
      {"role": "system", "content": "Answer in one sentence."},
      {"role": "user", "content": "Why does the first request take longer?"}
    ],
    "temperature": 0.2
  }'

The first call is slower because the backend loads the model on demand; subsequent calls hit a warm process. If the first call times out rather than being slow, raise your client’s timeout before you go looking for a bug — a cold load of a several-gigabyte model can outlast a default 30-second HTTP timeout comfortably.

The model YAML, and why it matters

Each model gets a YAML file in the models directory that names the backend, the weights file, the context size, the threads and the prompt template. Installing from the gallery writes one for you; that file, not the request, is where most behaviour is decided.

This is why the same GGUF can behave differently under LocalAI and under a desktop app. The template in the YAML is what turns your messages array into the string the model actually sees, and a template that does not match the model’s training produces output that is fluent and wrong-shaped — ignored system prompts, missing stop tokens, roles bleeding into the answer. When a model works everywhere else and not here, compare the template before you compare anything else. The context size in the same file is the other frequent culprit: a value set for a small default will truncate long prompts regardless of what the model supports.

The YAML is also where a model gets its public name. The name field is what appears in /v1/models and what clients put in the model field of a request, and it does not have to match the file on disk. That is more useful than it sounds: you can name a local model after the hosted one it stands in for, point an existing client at your endpoint with no code change at all, and swap the weights underneath later without touching the client. It is also how you end up confused about which weights answered a request, so keep the mapping written down somewhere other than the YAML.