Migrating Chunk Size Strategy When You Change Embedding Models

Nothing errors. The ingest job runs, the vectors land, the search endpoint returns results — and the results are worse. A chunk size is a tuned parameter of a specific embedding model, and swapping the model without revisiting it is the most common way a retrieval quality regression enters a system with no failing test to show for it.

Three limits, not one

“How long can a chunk be” has three different answers and people conflate them. Each moves independently when you change model.

• The hard input cap. The number of tokens the endpoint accepts. Exceed it and you get either a rejection or a silent truncation. OpenAI documents an 8,191-token input limit for its text-embedding-3 family; many models published through the sentence-transformers ecosystem have a max_seq_length of 256 or 512 word pieces, which is more than an order of magnitude smaller. Moving between those two worlds without changing chunk size means most of every chunk is discarded. OpenAI’s embeddings guide and the Sentence-Transformers documentation are the primary sources; check the model card for the specific checkpoint rather than assuming the family default.
• The effective context. The length beyond which retrieval quality falls even though the input is accepted. This is set by what the model was trained and evaluated on — typically passages rather than documents. A model whose training pairs were single paragraphs does not represent a two-page chunk well merely because it will accept one.
• The generation budget. Whatever your completion step can afford to paste in, which is the arithmetic in auditing token budget assumptions. This is the limit that changes when you swap the completion model, and it is easy to change one and reason about the other.

Why long chunks get blurry

The mechanism behind the effective-context limit is worth understanding, because it tells you which direction to move.

Most text embedding models produce one vector for the whole input by pooling over token representations — commonly a mean, sometimes a designated classification token. A mean is an average, and an average over a passage covering four topics sits somewhere between all four and is close to none of them. A query about the third topic has to compete against that blended vector, and it loses to a shorter chunk that is about only the third topic.

That gives the practical rule: chunk on semantic boundaries and keep each chunk about one thing, then let the token budget be an upper bound rather than a target. A chunker that fills to exactly 512 tokens every time is optimising the wrong quantity. It is also the reason that increasing chunk size to “give the model more context” usually helps generation and hurts retrieval — the two steps want different sizes, which is what small-chunk-retrieve, large-chunk-generate strategies exist to reconcile.

A second, separate effect: the tokenizer changed too. Two embedding models with the same stated cap of 512 tokens do not fit the same amount of your text if their vocabularies differ, and the gap is largest for non-Latin scripts and code. If your chunker is sized in characters, the effective chunk length in tokens moved even where the documented cap did not.

Re-deriving the size

Set the ceiling from the model, then choose the working size empirically against a query set you already have.

ceiling   = min(model_max_input_tokens, effective_context_estimate)
target    = a size at or below the ceiling, chosen by evaluation
overlap o = typically 0.10-0.20 * target, and o must be < target

# how many chunks a document of N tokens produces
stride  = target - o
chunks  = ceil( max(N - target, 0) / stride ) + 1

# worked: N = 10000, target = 512, o = 64
stride  = 448
chunks  = ceil( (10000 - 512) / 448 ) + 1 = ceil(21.18) + 1 = 23

# the same document at target = 256, o = 32
stride  = 224
chunks  = ceil( (10000 - 256) / 224 ) + 1 = ceil(43.5) + 1 = 45

Halving the chunk size roughly doubles the vector count, which doubles index storage and the per-query candidate set. That is the real cost of moving to a model with a shorter effective context, and it is usually larger than the embedding bill itself. The storage side is worked through in vector storage cost, and the dimension of the new model multiplies it again — a change of embedding dimension from 768 to 3072 is a fourfold increase in bytes per vector on top of any change in vector count.

Do not pick the target by argument. Take the queries out of your logs, build a small judged relevance set once, and sweep three or four candidate sizes through it. The sweep costs one afternoon and settles a question that otherwise gets re-litigated every quarter.

Costing the re-index

Every figure here is a placeholder you substitute from your own corpus and your provider’s current rate card; the point is the shape of the arithmetic, which does not change.

D  = documents in the corpus
Nd = mean tokens per document (measured with the NEW model's tokenizer)
c  = target chunk size,  o = overlap,  stride = c - o
p  = price per input token for the new embedding model

tokens_per_doc  = Nd * (c / stride)        # overlap is re-embedded, hence > Nd
total_tokens    = D * tokens_per_doc
embedding_cost  = total_tokens * p

vectors         = D * (Nd / stride)
storage_bytes   = vectors * dimension * bytes_per_component

# sensitivity: cost scales with c / (c - o).
#   c = 512, o = 64   -> 1.14x the raw corpus
#   c = 256, o = 64   -> 1.33x
#   c = 128, o = 64   -> 2.00x
# small chunks with a fixed overlap are dominated by the overlap.

The last block is the part people are surprised by. Overlap expressed as a constant number of tokens becomes an enormous proportion of the bill as chunks get smaller. Express overlap as a fraction of the target and it stays proportionate as you sweep.

You cannot mix two index generations

Vectors from two different embedding models are not comparable. They may not even have the same dimension, and where they do, the spaces are unrelated — a cosine similarity between them is a number with no meaning. There is no incremental migration where old documents keep their old vectors.

1. Build the new index alongside the old under a distinct name that includes the model identifier and the chunking parameters. The name is the record of what produced it.
2. Re-embed the whole corpus into it. Keep the chunk text and the chunk-to-document mapping in your own store rather than only in the vector database, so the next migration is a re-embed and not a re-ingest.
3. Run both indexes against the judged query set and compare recall at the k you actually retrieve, not at k = 100.
4. Cut over behind a flag, keep the old index readable for a rollback window, and only then delete it. The pattern and its pitfalls are covered in vector table migrations.