Picking an Embedding Model That Fits in 8GB of RAM

The model that will not load is rarely the problem on a small machine. What fails is the third batch, after a document longer than the others arrives, and the arithmetic that predicts it has nothing to do with parameter count.

What 8GB actually is

Start by subtracting what is gone before your process begins. These are planning assumptions, labelled as such, and you should replace them with what your own machine reports:

total                                     8.00 GB
- OS and background services         ~1.5-2.5 GB
- Python + torch + CUDA stubs, resident   ~0.7-1.0 GB
- headroom before the OOM killer          ~0.5 GB
= working budget                       ~4.0-5.3 GB

Call it 5 GB and treat that as the number the rest of this page spends. The PyTorch import alone is several hundred megabytes of resident memory before a single weight is read; on a machine this size, running under onnxruntime instead of PyTorch is worth roughly half a gigabyte of budget on its own, which is a large fraction of the difference between a base and a large model.

The weights, which are the small part

Weight footprint is parameters times bytes per parameter: 4 at fp32, 2 at fp16, 1 at int8. Using the published parameter counts:

model                        params    fp32      fp16     int8
bge-small-en-v1.5             33.4M   134 MB    67 MB    33 MB
bge-base-en-v1.5               110M   440 MB   220 MB   110 MB
nomic-embed-text-v1.5         ~137M   548 MB   274 MB   137 MB
gte-base-en-v1.5               137M   548 MB   274 MB   137 MB
bge-large-en-v1.5              335M  1.34 GB   670 MB   335 MB
multilingual-e5-base           278M  1.11 GB   556 MB   278 MB
multilingual-e5-large         ~560M  2.24 GB  1.12 GB   560 MB

Parameter counts are from the respective model cards; the byte columns are arithmetic. The e5-large figure is derived rather than published — 24 layers at width 1024 gives 24 x 12 x 1024² = 302M of transformer weights plus a 250,002 x 1024 embedding table at 256M, totalling 558M, which is why it is written as ~560M.

Against a 5 GB budget, every one of these loads with room to spare. Even the largest takes 45% of the budget at fp32 and 22% at fp16. Which is the point: if weights were the binding constraint, this would be a very short page.

Activations, which are not

A forward pass holds intermediate tensors, and their size depends on batch size and sequence length rather than on the model file. Two terms matter.

The hidden states are linear in everything: batch x sequence x width x 4 bytes, held several times over per layer while a layer is executing. For bge-base at batch 32 and 512 tokens that is 32 x 512 x 768 x 4 = 50 MB per tensor, so a few hundred megabytes live at once. Manageable.

The attention score matrix is the one that is quadratic in sequence length, and if the implementation materialises it:

batch x heads x seq^2 x 4 bytes

bge-base, batch 32, seq 512:
  32 x 12 x 512^2 x 4 = 402 MB     per layer, live at once

bge-base, batch 32, seq 1024 (hypothetically):
  32 x 12 x 1024^2 x 4 = 1.61 GB

nomic-embed, batch 8, seq 8192:
  8 x 12 x 8192^2 x 4 = 25.8 GB

That last line is why long-context embedding models require a memory-efficient attention kernel rather than merely benefiting from one. Flash-attention-style implementations never materialise the full matrix — they compute it in tiles — which turns the term from quadratic in memory to linear. Whether you get that depends on your runtime: PyTorch’s scaled dot-product attention selects an efficient kernel where it can, and an older path or an ONNX graph exported without it may not.

The rule this produces is the useful takeaway. Your peak memory is driven by batch x seq², not by batch alone. Doubling batch size doubles it; doubling sequence length quadruples it. A pipeline tuned on 200-token chunks that meets one 2,000-token document has just multiplied that term by a hundred.

Batching by tokens, not by documents

Fixed batch sizes are the direct cause of the intermittent OOM, because a batch of 32 short documents and a batch of 32 long ones are different workloads with the same name. Batch by token budget instead:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("BAAI/bge-base-en-v1.5")
TOKEN_BUDGET = 8192          # batch_size x padded_length, tuned once

lengths = [(len(tok.encode(t, truncation=True, max_length=512)), t)
           for t in texts]
lengths.sort(key=lambda p: p[0])          # sort so batches are uniform

batch, longest, out = [], 0, []
for n, t in lengths:
    if batch and (len(batch) + 1) * max(longest, n) > TOKEN_BUDGET:
        out.extend(model.encode(batch, normalize_embeddings=True))
        batch, longest = [], 0
    batch.append(t)
    longest = max(longest, n)
if batch:
    out.extend(model.encode(batch, normalize_embeddings=True))

This does two things at once. It caps the memory term at a constant you chose, so peak usage stops depending on what the corpus happens to contain. And because it sorts by length first, each batch is padded to nearly its own members’ length instead of to the longest document in the corpus, which removes the wasted computation that dominates the throughput arithmetic. One change, both problems.

The output order no longer matches the input order after sorting, so carry the original indices through and reorder at the end. Getting that wrong produces an index where every vector is attached to the wrong document, which is a failure that looks exactly like a bad model.

What fits

• English, 8 GB, comfortable. bge-base-en-v1.5 at fp32: 440 MB of weights plus roughly 0.5 GB of activations at a token budget of 8192, so about 1 GB of the 5 GB budget. Room for the rest of your application.
• English, 8 GB, tight machine. bge-small-en-v1.5 at 134 MB, 384 dimensions. A third of the index size of base as well, which on a million documents is 1.54 GB against 3.07 GB at fp32.
• Multilingual, 8 GB. multilingual-e5-base at 1.11 GB fp32 is the sensible ceiling. multilingual-e5-large at 2.24 GB loads but leaves under 3 GB for everything else, which forces a small token budget and therefore poor throughput — the trade is quality against documents per hour, and on a machine this size the base model usually wins on total work done.
• Long context, 8 GB. nomic-embed at 548 MB fp32 is fine; the risk is entirely the attention term. Confirm your runtime uses memory-efficient attention before you send it an 8,000-token document, or cap the sequence length well below the maximum.
• Out of reach. The 7B-parameter instruction-tuned embedding models are 28 GB at fp32, 14 at fp16 and 7 at int8. Even int8 exceeds the working budget before a single activation is allocated. These are not 8 GB models under any quantization.

One last term people forget: the index is not free either. A million 768-dimensional fp32 vectors is 1,000,000 x 768 x 4 = 3.07 GB, which on its own is most of the budget. If the index and the model must share a machine, either store the vectors on disk through something memory-mapped, or quantize them — int8 takes that 3.07 GB to 768 MB and binary to 96 MB, with a documented and measurable quality cost.