Sentence Transformers for Embeddings You Control

Sentence Transformers is the shortest path from text to a usable vector, and the two things it most needs you to get right are not about the model at all: whether the vectors are normalised, and whether queries and documents were encoded the way the model expects. Both fail silently, and both make retrieval worse in ways no exception reports.

What the library adds over a raw model

A transformer produces one vector per token. Turning that into one vector per sentence requires a pooling step — usually a mean over tokens, sometimes the first token’s output — and the correct choice is a property of how the model was trained. Get it wrong and you get vectors that are numerically fine and semantically useless.

This library packages the model, its pooling configuration and any normalisation layer together, so loading a model by name gives you something whose encode method returns the vector its authors intended. That is the whole value proposition, and it is worth more than it sounds.

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(MODEL_NAME)
vectors = model.encode(
    ["first sentence", "second sentence"],
    normalize_embeddings=True,
    batch_size=64,
)
vectors.shape       # (2, dimension)

Encoding, normalising, batching

Three arguments do most of the work.

• Normalisation. With unit-length vectors, a dot product is cosine similarity, which is what nearly every vector store computes by default. Normalise at encode time and store normalised vectors; then the store’s metric and your similarity function agree. Mixing normalised and unnormalised vectors in one index gives ranked results that are wrong without being empty, which is the hardest kind of bug to notice. Background: similarity metrics.
• Batch size. The main throughput lever. Larger batches use the GPU better up to the point of running out of memory, and the optimum depends on text length far more than on the model. Start at 32 or 64 and raise it while watching memory.
• Sequence length. Every model has a maximum input length, and text beyond it is truncated silently. Many popular embedding models cap at a few hundred tokens, which is shorter than people assume — encoding a two-page document produces a vector for its first paragraph and nothing else. Check the limit, then chunk to fit it rather than discovering the truncation through poor recall.

The prefix problem

A large share of embedding models are trained asymmetrically: queries and documents are encoded with different instruction prefixes, and the model card specifies them. Encoding both sides identically when the model expects prefixes measurably degrades retrieval, and produces no error of any kind.

The library has a mechanism for this — prompts registered with the model and selected by name at encode time — and the model’s own configuration may define the correct ones. The rule to follow regardless of the mechanism’s current name: read the model card, find out whether it wants asymmetric encoding, and if it does, encode your corpus one way and your queries the other, consistently, forever.

The asymmetry itself is not an implementation quirk; it reflects that a short question and a long passage are different kinds of text. The reasoning is in asymmetric embeddings.

What the vectors cost to store

Worth calculating before choosing a dimension, because it decides both your storage bill and your search latency.

bytes = n_vectors × dimensions × bytes_per_element

  float32 = 4 bytes    float16 = 2    int8 = 1    binary = 1/8

  1,000,000 chunks at 768 dimensions, float32
    1e6 × 768 × 4 = 3.07 GB

  same corpus at 384 dimensions, float32
    1e6 × 384 × 4 = 1.54 GB

  same corpus at 768 dimensions, int8
    1e6 × 768 × 1 = 0.77 GB

Index structures add overhead on top — typically tens of percent for
graph-based indexes, which also want the vectors resident in memory.

Two levers follow. Some models are trained so that a truncated prefix of the vector remains meaningful, which lets you cut dimensions without re-training — the mechanism behind Matryoshka embeddings. And quantising the stored vectors trades a little recall for a large reduction in memory, which is vector quantisation. Both are worth considering before buying more memory.

Fine-tuning on your own pairs

A general embedding model knows general similarity. If your domain treats two texts as related for reasons a general model cannot see — part numbers, internal jargon, a taxonomy nobody outside your company uses — fine-tuning on your own pairs is the highest-leverage change available, and it needs less data than people expect.

1. Collect positive pairs. Anchor and positive: a question and the passage that answers it, a search query and the document that was clicked, a ticket and the article that resolved it. A few thousand pairs is a real dataset; a few hundred is enough to try. Your logs usually already contain these.
2. Use a loss that manufactures negatives. The standard choice for pair data treats the other positives in the same batch as negatives, so you do not have to label anything as unrelated. Its consequence is that batch size is a quality parameter, not just a speed one — more in-batch negatives means a harder and more informative training signal.
3. Hold out a test set before training. Split by document or by customer, not randomly, or near-duplicates across the split will inflate the result.
4. Train briefly. One to three epochs at a small learning rate. Embedding fine-tuning overfits quickly, and the symptom is excellent test numbers with worse production retrieval.
5. Re-embed everything. A fine-tuned model produces vectors incompatible with the old ones. Budget the re-indexing cost and the cutover before you start, not after.

Proving it helped

Fine-tuning without measurement is a way to spend a week and gain confidence rather than quality. The measurement is not hard, and it is the same one you should already have for the untuned model.

Build a set of queries with known relevant documents, then compute recall at k and mean reciprocal rank for the base model and the tuned one on the same set. The library ships evaluator classes for exactly this, which can also run during training so you can see the curve rather than one final number.

Two honest caveats. Retrieval quality is not answer quality: a better recall number is a means, and the end is measured on the whole pipeline, which is the argument in RAG evaluation. And a reranker applied to a larger candidate set often beats a fine-tuned embedder for less effort — try reranking first, since it requires no re-indexing and can be removed if it does not help.

Serving embeddings in a request path

Encoding in a notebook and encoding inside a web request are different problems. Four differences account for nearly all of the trouble.

• Load the model once. Constructing the object reads weights from disk and takes seconds. It belongs at process startup, held as a module-level or dependency-injected singleton, never inside the handler. This is the single most common cause of an embedding endpoint that takes four seconds per call.
• Batch across requests, not only within them. A query is one short string, and encoding one string uses a fraction of the hardware. Collecting arrivals over a few milliseconds and encoding them together raises throughput substantially at a small, bounded latency cost — the same trade as continuous batching in text generation, at a much smaller scale.
• Cache by content hash. Query embeddings repeat far more than people expect, and a vector is a deterministic function of the text, the model and the prefix. Key a cache on a hash of all three — including the model identity, or a model change silently serves stale vectors.
• Bound the input. Encoding is quadratic in sequence length within the model’s limit, so a caller who posts a megabyte of text ties up the encoder. Truncate or reject at the edge, with a stated limit, rather than discovering it under load.

Whether to serve embeddings yourself at all is a separate question: a hosted embedding API removes the operational work and adds per-token cost and a dependency, while self-hosting is cheap at volume and pins your quality to a model you control. The comparison for a single GPU is in embedding models you can self-host, and the deciding number is usually how often you re-embed the whole corpus rather than the steady-state query rate.