Embedding Models You Can Self-Host

If you are going to self-host one part of a language model stack, make it the embeddings. The models are small, the workload batches perfectly, there is no autoregressive loop, and the thing you send them is usually the most sensitive data you have.

Why this is the easy one

An embedding model is an encoder. One forward pass over a chunk of text produces one vector, and that is the whole job — no token-by-token loop, no KV cache, no per-request state. Throughput is therefore bounded by arithmetic rather than by memory bandwidth, which is the regime GPUs are good at, and large batches are free in a way they never are for generation.

• The models are small. Most sit between roughly 100M and 1B parameters. Even at full precision that is a fraction of a card.
• The work is embarrassingly batchable. Indexing a corpus is a queue of independent items; you can saturate hardware with no scheduling cleverness at all.
• Latency requirements are asymmetric. Indexing is offline and cares only about throughput. Query embedding is one short string and is fast on anything, including a CPU.
• The privacy argument is strongest here. Building an index means sending your entire document corpus through the model, not just the fragments a user happens to ask about. If anything in your stack should not leave the building, it is this.
• Quality is far less size-dependent than for generation. Small embedding models are genuinely competitive with large ones, because the task is representation rather than reasoning.

Storage and throughput arithmetic

The model is not your capacity problem; the vectors are. Work this out before you choose a dimensionality, because it is the number that decides your infrastructure:

vector_bytes = n_chunks * dimensions * bytes_per_component

1M chunks, 1024 dims, fp32   1e6 * 1024 * 4 = 4.1 GB
1M chunks, 1024 dims, fp16   1e6 * 1024 * 2 = 2.0 GB
1M chunks,  384 dims, fp32   1e6 *  384 * 4 = 1.5 GB
10M chunks, 1024 dims, int8  1e7 * 1024 * 1 = 10.2 GB

# plus the index structure itself, commonly a further 20-50%
# for a graph-based index, and it wants to be in RAM.

Two levers, both underused. Quantise the vectors — int8 quantisation of embeddings typically costs very little retrieval quality and cuts storage fourfold, and binary quantisation goes further for a first-pass shortlist that a re-rank step then fixes. Use fewer dimensions — several modern embedding models are trained so that a truncated prefix of the vector remains usable, which lets you trade dimensions against quality after the fact rather than committing at index time. If your model supports that, it is the single most useful property it has.

For throughput, measure rather than estimate — it depends on chunk length, batch size and hardware in ways no published figure will match:

# time a realistic batch of your own chunks, not synthetic text
python - <<'PY'
import time, json, requests
chunks = [json.loads(l)["text"] for l in open("chunks.jsonl")][:2000]
t = time.time()
for i in range(0, len(chunks), 64):
    requests.post("http://localhost:8080/embed",
                  json={"inputs": chunks[i:i+64]}).raise_for_status()
d = time.time() - t
print(f"{len(chunks)/d:.0f} chunks/s -> {len(chunks)/d*3600/1e6:.2f} M chunks/hour")
PY

That last figure is the one that matters, because it tells you how long a full re-index takes — and re-indexing is not hypothetical, as the last section explains.

Serving them

# a dedicated embedding server, batching and warm
docker run --gpus all -p 8080:80 \
  ghcr.io/huggingface/text-embeddings-inference:latest \
  --model-id org/embedding-model \
  --max-batch-tokens 16384

# or in-process, which is fine for indexing jobs
from sentence_transformers import SentenceTransformer
m = SentenceTransformer("org/embedding-model", device="cuda")
v = m.encode(chunks, batch_size=64, normalize_embeddings=True,
             convert_to_numpy=True)

Two details that cause more retrieval bugs than model choice ever does. First, asymmetric models need their prefixes: many embedding models are trained with distinct instructions for queries and for documents, and omitting them on one side silently degrades every result. Read the model card and apply exactly what it specifies. Second, normalise consistently — if you normalise vectors at index time you must normalise at query time, and your distance metric must match. Cosine similarity on unnormalised vectors is a bug that produces plausible-looking rankings.

Choosing on your own corpus

Public embedding benchmarks are useful for building a shortlist and nearly useless as a decision, because retrieval quality is corpus-specific in a way that generation quality is not. A model trained on general web text may do poorly on your legal filings or your telemetry logs regardless of its rank.

• Shortlist from a benchmark filtered to your language, your dimensionality budget and a permissive licence. Three or four candidates.
• Build a small labelled set. Fifty to two hundred real queries, each with the chunk or chunks that should be retrieved. Mining these from your search logs or support tickets takes an afternoon and is the highest-value hour in the project.
• Measure recall at k — the fraction of queries where a correct chunk appears in the top k, for the k your pipeline actually passes to the model. This is the metric that predicts downstream answer quality; raw similarity scores do not.
• Hold the chunking fixed across candidates. Chunk size and overlap frequently matter more than the model, so varying both at once tells you nothing.
• Test a re-ranker separately. A cross-encoder re-ranking the top 50 down to the top 5 often buys more than any embedding upgrade, and it is a different, cheap decision.

The cost nobody budgets for

Embeddings from different models are not comparable. Not approximately, not with a transformation — a vector from one model has no meaningful relationship to a vector from another. So changing embedding model means re-embedding the entire corpus and rebuilding the index, and at ten million chunks that is a real job with a real bill.

This makes the initial choice unusually sticky, and it argues for a few habits that cost nothing now:

• Store the model identifier and revision alongside every vector. Mixed-provenance indexes are silently broken and very hard to diagnose.
• Keep the chunk text. Re-embedding from stored text is a job; re-extracting from source documents is a project.
• Prefer a permissively licensed model you can pin. A self-hosted model cannot be deprecated underneath you, which for an artefact this expensive to rebuild is worth more than a small quality edge.
• Version the index. Build the new one alongside the old, compare on the labelled set, then switch. Never re-embed in place.