Fine-Tuning an Embedding Model on Your Own Data

Fine-tuning an embedding model is unusually cheap — a base model of a hundred million parameters, fifty thousand pairs and an hour on one GPU. What is not cheap is everything that happens after: a fine-tuned model is a new model, and every vector you have stored becomes incompatible on the day you deploy it.

When this is worth doing

Fine-tuning pays when your domain’s notion of relevance differs from the general one the base model learned. Concretely: your users search with vocabulary the model has never associated with your documents (internal jargon, product codenames, an industry’s terms of art), or your relevance judgements are idiosyncratic in a way no prompt captures — a legal team for whom two superficially similar clauses are not interchangeable, a parts catalogue where the difference between two fittings is the entire point.

It does not pay when retrieval is failing for a reason fine-tuning cannot fix, and it usually is. Bad chunking, missing documents, no lexical index for identifier queries, a query prefix applied on one side and not the other — all of these look like “the model doesn’t understand our domain” and none of them are. Exhaust the cheap fixes first; they are also the reversible ones.

Mining pairs from click logs

The training data is pairs of (query, relevant document). Click logs contain them, mixed with a great deal of noise, and the mixing is systematic rather than random — which means it can be corrected.

The dominant distortion is position bias: users click the first result because it is first, not because it is best. Train on raw click data and you teach the model to reproduce your existing ranker, including its mistakes, which is an expensive way to change nothing. Three filters do most of the correction:

• Prefer clicks below position one. A click on result 5 is much stronger evidence than a click on result 1, because the user passed over four alternatives to reach it.
• Require a satisfied signal. Dwell time past some threshold, or the click being the last in the session — a click followed by an immediate return to the results page is evidence of the opposite of relevance.
• Drop navigational queries. Queries that always produce the same click teach the model nothing and, in bulk, will dominate the loss.

Aim for a few thousand pairs to see whether the approach helps at all, and tens of thousands for a model you would ship. If you have no click logs, generating synthetic queries from documents with a language model is a legitimate starting point — one or two questions per chunk, then a filtering pass that discards any whose gold chunk the base model already retrieves at rank 1, since those pairs carry no gradient worth having.

The loss, and why batch size is a hyperparameter

The standard objective — Sentence Transformers calls it multiple negatives ranking loss, the literature calls it InfoNCE — needs only positive pairs. Every other document in the batch serves as a negative:

batch of B pairs (q_i, d_i)

S = scale * cosine(Q, D)          # (B, B) matrix, scale ~ 20
loss = cross_entropy(S, labels=[0, 1, 2, ..., B-1])

# row i: the correct answer is column i; the other B-1 columns
# are negatives you got for free

This is why batch size is not merely a memory setting here. A batch of 16 asks the model to pick the right document out of 16; a batch of 256 asks it to pick out of 256, which is a far harder and more useful task. Larger batches genuinely improve the resulting model, and gradient caching or in-batch-negative sharing across devices exists specifically to get them. The scale factor is the inverse temperature on the softmax and 20 is the conventional default; it sharpens the distribution so that near-misses are penalised meaningfully.

One data hazard follows directly from the loss: if two rows in a batch happen to be near-duplicates, each is used as a negative for the other and the model is trained on a contradiction. Deduplicate the training pairs before you start.

Hard negatives

In-batch negatives are random and therefore easy — most randomly chosen documents are obviously irrelevant, and the model learns to separate them quickly and then stops improving. Hard negatives are documents that are plausibly relevant and are not, and they are where the remaining quality lives.

for (query, positive) in pairs:
    top = base_model.retrieve(query, k=50)
    hard = [d for d in top[10:50] if d != positive][:4]
    emit (query, positive, hard)

Note the top[10:50]. The highest-ranked non-positive results are very often actually relevant documents that simply were not the labelled one, and training on those as negatives teaches the model that correct answers are wrong — a well-documented way to make a fine-tune worse than the base model. Skipping the top ten is a crude but effective guard. If you have the budget, an even better filter is to have a strong reranker score the candidates and discard any it rates highly.

The training run

from sentence_transformers import SentenceTransformer, losses, InputExample
from torch.utils.data import DataLoader

model = SentenceTransformer("BAAI/bge-base-en-v1.5")

examples = [InputExample(texts=[q, pos, neg]) for q, pos, neg in triples]
loader   = DataLoader(examples, batch_size=64, shuffle=True, drop_last=True)
loss     = losses.MultipleNegativesRankingLoss(model)

model.fit(
    train_objectives=[(loader, loss)],
    epochs=3,
    warmup_steps=int(0.1 * len(loader) * 3),
    optimizer_params={"lr": 2e-5},
)

Scale of the job: 50,000 triples at batch 64 for 3 epochs is 50000 × 3 / 64 ≈ 2,344 steps. On a single modern GPU with a 110M-parameter base model that is minutes to a couple of hours — this is genuinely one of the cheapest useful fine-tunes in machine learning, and the reason to hesitate is never the training cost.

Two guardrails. Use a low learning rate (2e-5 is the conventional starting point) and few epochs; embedding models overfit fast and a model that has memorised your training queries retrieves beautifully for them and worse than the base model for everything else. And always evaluate against the base model on a held-out set from the start — “the fine-tune is worse” is a common and completely normal outcome that you want to discover in an hour, not after re-indexing.

Shipping it, which is the expensive half

A fine-tuned model produces vectors in a different space from its own base model. Not slightly different — incompatible, in exactly the way two unrelated models are. Deploying it means re-embedding the entire corpus, running the dual-column migration with all its steps, and doing that again for every subsequent fine-tune.

That reframes the decision. The training is an hour; the shipping is a project, and you will want to ship more than once as you gather more click data. Two things make it bearable: batch your fine-tunes on a schedule rather than deploying each experiment, and keep the evaluation harness good enough that most candidates are rejected before they reach the migration. The gold set is, once again, the asset.

Finally, run the fixed-string canary against your own fine-tuned checkpoint too. A self-hosted model is a file that somebody can replace, and the failure looks identical to a provider silently changing a hosted one.