Running GTE Embedding Models Locally

GTE v1.5 looks like another BERT-sized embedding model with a bigger context number in the specification table. It is not: three components of the encoder were replaced, and each substitution has a consequence you can observe when you serve it.

The backbone is not BERT

Alibaba describes the GTE v1.5 encoder as a transformer++ backbone — BERT with rotary position embeddings and gated linear units. The Alibaba-NLP/gte-base-en-v1.5 model card states the architecture and the family sizes: gte-base-en-v1.5 at 137M parameters, 768 dimensions and 8192 tokens; gte-large-en-v1.5 at 434M, 1024 dimensions and the same context; and a much larger instruction-tuned Qwen-based model at 7.7B and 4096 dimensions. All Apache-2.0.

• Rotary embeddings instead of learned absolute positions. Original BERT has a fixed table of 512 learned position vectors and physically cannot address position 513. Rotary embeddings encode position by rotating the query and key vectors as a function of index, so position is a function rather than a lookup and there is no table to run off the end of.
• Gated linear units in the feed-forward block. A gated block splits the up-projection into a value path and a gate, multiplies them, and projects back. It costs a third more parameters per layer at the same width for a consistently better quality trade, which is why almost every model designed after 2021 uses one.
• Memory-efficient attention. The card recommends enabling xformers, which computes attention without materialising the full sequence-by-sequence score matrix. At 8192 positions that matrix would be 67 million entries per head per layer; not materialising it is the difference between the long context being usable and being theoretical.

What 8192 tokens buys and does not

Rotary embeddings make long positions representable. They do not by themselves make them good: a model still has to be trained at length for its behaviour there to be reliable, and quality at 8000 tokens is not automatically the quality you see at 500.

There is also a retrieval argument against using the whole window that is independent of the model. One vector must summarise everything you put into it, and the summary of a document covering nine topics sits near none of the nine. Long-context embedding models are most useful for documents that are genuinely about one thing and were previously being split mid-argument — a contract clause, a single support article, one function with its docstring. They are not a licence to stop chunking.

The cost side is quadratic in sequence length for the attention term. Going from 512 to 8192 tokens is 16x the tokens and 256x the attention work, offset only by whatever the memory-efficient kernel saves in bandwidth. Per document, embedding at full length is far more expensive than the 137M parameter count suggests.

CLS pooling and no instruction

GTE pools from the first token: the model card’s own snippet reads the embedding as outputs.last_hidden_state[:, 0]. That puts it with BGE and against E5 and Nomic, and it is the detail that silently degrades a system where somebody swapped one model for another and left the pooling code alone. If you are serving GTE through anything that lets you choose — llama.cpp’s --pooling flag, for instance — the correct value is cls, and getting that flag wrong produces plausible vectors that rank badly.

Unlike BGE and E5, the base GTE English models do not document a required prefix or instruction. Encode the text as it is. The instruction-tuned members of the family — the Qwen-based ones — do take an instruction, which is a property of those specific models rather than of the family name, so read the card of the exact checkpoint you are loading.

Loading requires trust_remote_code=True, because the transformer++ backbone is not a class shipped inside transformers. As with any model that ships its own modelling code, that means executing Python from a model repository in your process, and it means an offline deployment has to cache the code as well as the weights.

Unpadding, and why batching gets cheaper

The card recommends enabling unpadding alongside memory-efficient attention, and this is the optimisation with the largest practical effect on throughput for a mixed corpus.

Ordinarily a batch is padded to its longest member, and every padded position costs a full forward pass through every layer before being masked out at the end. Batch a 20-token title with a 4000-token article and you compute 4000 positions twice. Unpadding concatenates the real tokens of the batch into one flat sequence with offsets, so only real tokens are processed.

Two consequences follow. First, the saving scales with how uneven your lengths are; on a corpus of near-identical chunk sizes it does almost nothing. Second, and more usefully, length-sorted batching is worth doing even without unpadding: sort your corpus by token count before batching so that each batch contains similar lengths, and the wasted padding collapses. That costs one sort and no special kernel, and it is the version of this optimisation available to every model in this cluster.

The quantized variant

Quantized GTE checkpoints — typically int8 ONNX exports — change three things, and it is worth separating them because they are often conflated.

• The weights get smaller. int8 is a quarter of fp32, so gte-base goes from roughly 548 MB of weights (137M x 4 bytes) to roughly 137 MB. That is arithmetic, not a measurement.
• CPU throughput improves, GPU throughput mostly does not. On a CPU the forward pass at small batch is bound by reading weights from memory, so quarter-sized weights is close to a direct win. On a GPU with plenty of bandwidth the arithmetic is the bottleneck and int8 helps only where the kernels genuinely execute in int8.
• The output vectors move. This is the part that matters and the part nobody publishes a general number for. The quantized model produces slightly different vectors from the fp32 one, which means an index built with one and queried with the other is subtly wrong.

The rule that follows is absolute: never mix precisions across an index. Re-embed the whole corpus when you change the model’s numerics, exactly as you would when changing the model. How much drift to expect, and how to measure it on your own corpus rather than trusting a headline figure, is a derivation of its own.