Serving Embeddings With llama.cpp

llama.cpp’s server will run an embedding model over HTTP with no Python anywhere in the deployment, which is a real advantage when the alternative is a PyTorch container. The catch is a single flag whose wrong value produces vectors of the correct shape and the wrong meaning.

Why serve embeddings this way

Three reasons, in descending order of how often they apply.

• One runtime. If you are already running a GGUF chat model through llama-server, adding embeddings means a second process of the same binary rather than a Python service with its own dependency tree, CUDA build and container image.
• Deployment weight. A llama.cpp binary plus a quantized GGUF is tens of megabytes plus the model. The equivalent sentence-transformers image is measured in gigabytes, most of it PyTorch.
• CPU quantization for free. GGUF carries its quantization in the file, so a q8_0 embedding model needs no export step and no ONNX toolchain — compare the ONNX route, which gets you further on optimisation but asks for more setup.

What you give up is the ecosystem: no encode() convenience, no automatic pooling configuration read from the model repository, and no Matryoshka helpers. Every convention the Python library applied for you becomes a flag you must set correctly.

Starting the server

llama-server \
  -m models/bge-base-en-v1.5-q8_0.gguf \
  --embeddings \
  --pooling cls \
  -c 512 \
  -ub 512 \
  --host 127.0.0.1 --port 8081

--embeddings (accepted as --embedding) puts the server into embedding mode. The llama.cpp server documentation describes this as restricting the server to the embedding use case and says to use it only with dedicated embedding models — which is the right reading: a server in this mode is not also going to serve chat completions, so a hybrid deployment is two processes.

-c is the context size and should match the model’s training length: 512 for BGE and E5, 8192 for Nomic and GTE v1.5. Setting it higher than the model supports does not extend the model; setting it lower silently truncates your inputs.

The pooling flag is the whole game

--pooling takes none, mean, cls, last or rank, and the server documentation states that the model default is used if it is unspecified. That default comes from metadata baked into the GGUF at conversion time, and whether it is right depends on whoever converted the file — which is exactly the kind of thing to verify rather than assume.

The correct value is a property of how the model was trained:

• BGE, GTE v1.5 — cls. Trained reading the first token’s final hidden state.
• E5, Nomic Embed — mean. Trained on a masked average over the token vectors.
• Decoder-based embedding models — often last, since a causal model’s final position is the only one that has seen the whole input.
• none — returns one vector per token rather than one per input. Useful for late-interaction retrieval, and not what you want otherwise. The OpenAI-compatible endpoint requires a pooling type other than none.

Set it wrong and nothing errors. You get a float array of the right length, cosine similarities in a plausible range, and a retrieval system that is meaningfully worse than it should be with no signal telling you why. The check is to embed a handful of texts through both this server and sentence-transformers with the same model, and confirm the cosine between the two vectors for each text is essentially 1.0. If it is 0.8, the pooling does not match.

Two endpoints, two shapes

The server exposes an OpenAI-compatible endpoint and a native one. The compatible one is what you want if any existing client already speaks that shape:

curl -s http://127.0.0.1:8081/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": ["the cat sat on the mat", "a feline rested on a rug"]}' \
  | jq '.data[0].embedding | length'

The native endpoint takes content rather than input and accepts an embd_normalize parameter controlling normalisation:

curl -s http://127.0.0.1:8081/embedding \
  -H "Content-Type: application/json" \
  -d '{"content": "the cat sat on the mat"}'

For any pooling type other than none, the server normalises with the Euclidean norm, so the vectors come back as unit vectors and a dot product is already cosine similarity. Do not normalise again in your client; it is harmless but it hides whether the server did.

One thing the server will not do for you: apply the model’s prefix convention. A BGE query still needs its instruction, an E5 input still needs "query: " or "passage: ", and Nomic still needs one of its four task prefixes. Those live in your client code now, because there is no Python layer left to hold them.

Batch size and long inputs

Pooling requires the whole sequence to be present in one physical batch, since you cannot average over tokens that were processed in separate passes. That makes -ub / --ubatch-size — the physical batch size, documented with a default of 512 — a correctness setting rather than only a performance one for embedding workloads. If your longest input is 8192 tokens, the physical batch has to be able to hold it.

# long-context embedding model
llama-server -m models/nomic-embed-text-v1.5.f16.gguf \
  --embeddings --pooling mean -c 8192 -ub 8192

Raising -ub costs memory in proportion, because the KV and activation buffers are sized from it. The efficient arrangement for a bulk job is therefore to keep -ub just above your longest chunk, and to send many short texts per request so the server can pack them.

1. Get a GGUF embedding model, or convert one with convert_hf_to_gguf.py from the llama.cpp repository.
2. Start llama-server with --embeddings, an explicit --pooling matching the model, and -c / -ub at the model’s context length.
3. curl the /v1/embeddings endpoint and confirm the vector length matches the model’s published dimension.
4. Cross-check three texts against sentence-transformers; cosine between the two implementations should be ~1.0. If not, fix the pooling before you index anything.