Running an MLX Model as a Local OpenAI-Shaped API

mlx-lm ships a small HTTP server that speaks the OpenAI chat completions shape, which means every client library you already have can talk to a model running on your own machine by changing a base URL. Getting there is one command; knowing what you have got is the rest of this page.

Start it

1. pip install mlx-lm on a native arm64 Python, macOS 14.0 or newer.
2. Start the server against a model, by Hub id or local path:

   mlx_lm.server --model mlx-community/Qwen3-14B-4bit --port 8080

3. Confirm it is up before you debug anything else: curl http://127.0.0.1:8080/health.

The defaults are --host 127.0.0.1 and --port 8080. The host default is the important one: out of the box the server is bound to loopback and is not reachable from your network. Read the What it is not section below before you change that.

The first request

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mlx-community/Qwen3-14B-4bit",
    "messages": [
      {"role": "system", "content": "Answer in one sentence."},
      {"role": "user", "content": "What limits decode speed on a Mac?"}
    ],
    "max_tokens": 200,
    "temperature": 0.7
  }'

The response is the familiar object with a choices array, a message with a role and content, a finish_reason and a usage block. Add "stream": true and you get server-sent events with the usual delta chunks, terminated by data: [DONE].

Because the shape matches, the official OpenAI Python client works against it directly:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-used")

resp = client.chat.completions.create(
    model="mlx-community/Qwen3-14B-4bit",
    messages=[{"role": "user", "content": "What limits decode speed on a Mac?"}],
    max_tokens=200,
)
print(resp.choices[0].message.content)

The api_key is required by the client library and ignored by the server, which is worth understanding rather than pattern-matching past: there is no authentication here at all.

What it serves

• POST /v1/chat/completions — the message-array endpoint, which applies the model’s chat template for you.
• POST /v1/completions — the raw-text endpoint, no template applied.
• GET /v1/models — the model listing.
• GET /health — a liveness check, and the right thing for a supervisor or a shell script to poll.

There is no embeddings endpoint and no audio endpoint here; those are separate packages. A client library that assumes the full OpenAI surface will find the gaps at runtime rather than at configuration time.

Options worth setting

The defaults are chosen for a single interactive user. Four flags change that meaningfully:

• --max-tokens defaults to 512. A client that omits max_tokens gets that, and truncation with finish_reason of "length" is the most common “the model stopped mid-sentence” report against a local server.
• --temp, --top-p, --top-k, --min-p set the server-wide defaults for requests that do not specify their own. --temp defaults to 0.0 — deterministic argmax — which surprises people expecting a hosted provider’s default of around 1.0.
• --draft-model and --num-draft-tokens turn on speculative decoding with a smaller model of the same tokenizer family. It is the one setting that can raise single-request decode speed above what the bandwidth arithmetic in the unified-memory ceiling allows for the large model alone, because several tokens are verified per pass over the weights.
• --decode-concurrency and --prompt-concurrency default to 32 and 8, and govern how many batchable requests are decoded and prefilled together. Batching helps throughput on a machine whose bottleneck is bandwidth, because one pass over the weights serves several sequences — but it costs one KV cache per concurrent sequence, which is memory you may not have.

--chat-template-args takes a JSON string forwarded to apply_chat_template, which is how you disable a reasoning mode on the families that have one.

What it is not

The server has no authentication, no rate limiting, no request quotas and no tenancy. Binding it to 0.0.0.0 puts an unauthenticated endpoint that executes arbitrary prompts on your network. If you need it reachable from another machine, put it behind something that terminates TLS and checks a credential, or reach it over an SSH tunnel or a private overlay network. That is not paranoia about the model; it is that a local server is a general-purpose compute endpoint.

It is also one process holding one model resident. There is no model swapping, no queue you can inspect, and no eviction under memory pressure — a second large model means a second process and a second copy of the weights in the same unified pool. On the throughput side, batching exists but a Mac is not a serving platform; the general argument for when to run your own is in self-host versus API.

There is no persistence either. Restarting the process re-reads the weights from disk, which for a 40 GB model is tens of seconds before the first request is served, and the /health endpoint is the thing to gate on rather than a fixed sleep. If you are supervising it with launchd so it comes back after a crash, remember that a crash under memory pressure will be followed by a restart into the same memory pressure — a restart loop on a Mac that is quietly swapping looks identical to a healthy service that keeps briefly failing.

And the memory arithmetic does not go away because there is a server in front of it. Every concurrent sequence carries its own KV cache, so --decode-concurrency 32 on a model whose cache is 0.328 MB per token means thirty-two caches growing independently; at a few thousand tokens each that is gigabytes nobody budgeted for. Cap it to what you actually serve, and cap the context alongside it.