Running Nomic Embed Locally

Nomic Embed is an Apache-2.0 embedding model with an 8192-token context and truncatable dimensions, which makes it the obvious choice for a local index over long documents. It also has the most demanding prefix convention of the common families and a loading requirement that turns “run it offline” into a real step.

Four prefixes, not two

Where E5 has two roles, Nomic has four task prefixes, and the nomic-ai/nomic-embed-text-v1.5 model card documents what each is for:

• search_document: — text you are indexing, the corpus side of retrieval.
• search_query: — a question you want answered from that corpus.
• clustering: — grouping texts, discovering topics, removing semantic duplicates.
• classification: — producing vectors that will be features for a downstream classifier.

The split between clustering: and classification: is the one with no analogue elsewhere, and it is a real distinction rather than a nicety. Clustering wants a space where distance is meaningful in every direction. Classification wants a space where a linear boundary can separate labels, and those are not the same geometry. The model was trained with both objectives under their respective prefixes and will give you a different arrangement depending on which you ask for.

The practical consequence is that vectors produced under different prefixes are not interchangeable. A corpus embedded under clustering: for a deduplication pass cannot be reused as a retrieval index; that would need a second pass under search_document:. Budget for embedding the corpus twice if you need both, rather than discovering the mismatch through poor recall.

Running it

pip install "sentence-transformers>=3.0" einops

python - <<'PY'
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("nomic-ai/nomic-embed-text-v1.5",
                            trust_remote_code=True)

docs = ["search_document: " + d for d in [
    "Utrecht Centraal is the busiest railway station in the Netherlands.",
    "The Afsluitdijk was completed in 1932.",
]]
qs = ["search_query: which dutch station is busiest"]

D = model.encode(docs, normalize_embeddings=True)
Q = model.encode(qs, normalize_embeddings=True)
print(D.shape)             # (2, 768)
print((Q @ D.T).round(3))
PY

Note that the model card’s own quick-start snippet omits the prefixes for brevity. Do not copy that pattern into anything you index with. The model is 0.1B parameters, pools by masked mean, produces 768 dimensions natively, and is Apache-2.0 — genuinely permissive, unlike several models in this space whose weights carry a bespoke licence.

trust_remote_code and what offline means

trust_remote_code=True is required, and it is worth being precise about what it does. Nomic Embed’s architecture is not one of the classes built into transformers; the repository ships its own Python modelling code, and the loader downloads that code from the Hub and executes it in your process. You are running arbitrary Python from a model repository, which is a supply-chain decision rather than a checkbox.

It also complicates offline operation, because “offline” now means having both the weights and the code cached before the network goes away. Fetch everything once on a connected machine:

# on a connected machine
huggingface-cli download nomic-ai/nomic-embed-text-v1.5 \
  --local-dir ./nomic-embed-text-v1.5

# on the offline machine
export HF_HUB_OFFLINE=1
python -c "
from sentence_transformers import SentenceTransformer
m = SentenceTransformer('./nomic-embed-text-v1.5', trust_remote_code=True)
print(m.encode(['search_query: hello']).shape)
"

The download brings the weights, the tokenizer, the sentence- transformers configuration and the Python modules the architecture needs. Read those modules before you run them on a machine that matters — they are a few hundred lines and they are the actual code that will execute in your process. Reviewing them once and pinning the revision converts an ongoing trust decision into a one-time one, which is the only version of this that is defensible in a deployment where the isolation is the point.

Setting HF_HUB_OFFLINE=1 is what turns a silent network attempt into an immediate error, which is what you want on a machine that is supposed to be isolated — a hang on a socket timeout is a much worse way to find out. Pin the revision you downloaded if you care about reproducibility, because the remote code can change under a moving branch name.

Truncating the dimension

v1.5 was trained with Matryoshka representation learning, which means the information in the vector is ordered: the first 256 components carry more than a random 256 would. The card documents 768 natively with 512, 256, 128 and 64 as supported truncations.

Truncation is not just slicing. You slice, then re-normalise, because the prefix of a unit vector is not a unit vector:

import torch.nn.functional as F

full = model.encode(docs, convert_to_tensor=True)
short = F.normalize(full[:, :256], p=2, dim=1)

Skipping the re-normalisation gives you vectors whose norms vary with how much of their energy happened to sit in the first 256 dimensions, and dot-product search on those ranks by a mixture of similarity and norm. The arithmetic case for doing it at all is straightforward: at 256 dimensions your index is a third the size and your query is a third the arithmetic, for a quality loss the technique is specifically designed to keep small. On a million documents at fp32 that is 3.07 GB against 1.02 GB, which is often the difference between an index that stays in RAM and one that does not.

Truncate consistently or not at all. Every vector in an index must have the same dimension, and a query truncated to 256 cannot be compared against documents stored at 768 — the dot product is not even defined. That sounds obvious and is nonetheless the common way this feature goes wrong, because the truncation lives in application code rather than in the model, so a query path and an ingestion path written at different times can disagree. Put the target dimension in the same configuration object as the model name.

It is also worth being clear about what Matryoshka training does not give you. The ordering of information across dimensions is a property of the training objective, not a mathematical guarantee, so the quality curve as you truncate is empirical: 512 is close to free, 64 is not, and where your corpus sits between them is something to measure rather than assume. Embed a few hundred queries at each candidate dimension and compare top-10 overlap against the full-dimension result; the point where that overlap starts falling is your floor.

The 8192-token context

The headline feature is a context sixteen times longer than the 512-token BERT-based models. It removes a class of chunking problem — a full support article or a long meeting transcript fits in one vector — but it does not remove the reason to chunk.

A single vector for 8,000 tokens is an average of a great deal of material, and averaging is lossy in a specific way: a document about nine topics produces a vector near none of them. Retrieval quality on long documents usually gets worse as you embed more text per vector, even when the model can technically accept it. The long context is most useful when the document is genuinely about one thing and you were previously being forced to split it mid-argument.

The other cost is quadratic. Attention over 8192 positions is sixteen-squared — 256 times — the attention work of 512 positions, and the memory to match if the implementation materialises the attention matrix. Embedding long documents is dramatically more expensive per document than the parameter count suggests, which is the term that dominates the FLOPs-per-document arithmetic once sequences get long.

1. Download the repository once with huggingface-cli download, code and weights together.
2. Load from the local directory with trust_remote_code=True and HF_HUB_OFFLINE=1 set.
3. Pick one of the four task prefixes and apply it in exactly one place.
4. If you truncate, slice then re-normalise, and verify the norms are 1.0.