Writing a Regression Test for a Chunking Change

Changing chunk size looks like a tuning knob and behaves like a schema migration. Every vector in the index was computed from a chunk, and moving the boundaries changes what each vector means — including, occasionally, splitting the one sentence that answered a question into two halves that answer nothing.

What a chunking change actually breaks

The failure is specific and worth stating precisely. A question is answered by some span of text — a sentence, a table row plus its header, a numbered step plus its condition. If that span lies entirely within one chunk, retrieving that chunk answers the question. If a new boundary falls inside the span, neither resulting chunk contains the answer, and both are still plausible enough to be retrieved. The system returns confident, well-sourced, wrong output.

This is why the obvious assertions do not help. Chunk count changes by design. Average chunk size changes by design. Even mean retrieval score can improve while the specific spans you care about are being cut in half. The property to assert is the one that actually matters: for each known question, the answering span is still contained in a single chunk, and that chunk is still retrieved.

Spans, not chunk counts

The fixture is a list of questions, each with a verbatim substring that must survive intact. Take the substrings from the documents themselves — copy them, do not retype them — because the test is exact string containment and a re-typed span with a different apostrophe fails for the wrong reason.

# tests/fixtures/spans.py
SPANS = [
    {
        "query": "how long do I have to return something",
        "doc": "policy/returns.md",
        # Must survive inside ONE chunk: the window and the condition together.
        "span": "within 30 days of delivery, provided the item is unused",
    },
    {
        "query": "what is the fee for a late return",
        "doc": "policy/returns.md",
        # A table row is meaningless without its header.
        "span": "| Late (31-60 days) | 15% |",
    },
    {
        "query": "do I need a photo for a damaged item refund",
        "doc": "policy/refunds.md",
        "span": "Damaged-item refunds require a photo uploaded within 48 hours",
    },
]

Add a row every time retrieval gets something wrong in production and the cause turns out to be a boundary. That is how this fixture earns its keep: it is not a design document, it is a list of things that have gone wrong, and it grows in the direction of your actual failures.

Keep the fixture documents in the repository beside the spans, and keep them small — a few hundred lines each, cut down from the real thing but with the awkward structure intact. A span fixture pointing at documents that live in a content management system breaks the day an editor rewords a sentence, and the failure looks exactly like a chunking regression when it is nothing of the sort. If you must test against live documents, pin a revision.

Do not put expected chunk boundaries in the fixture. That is the trap version of this test: it pins the current implementation rather than the property you need, so every legitimate chunker change fails it and the fixture gets regenerated without thought. The span formulation is deliberately weaker than an exact boundary, and it is weaker in exactly the right place — any strategy that keeps the answer together passes, however different its boundaries are.

The test

Two assertions per row. The span is contained in exactly one chunk, and that chunk comes back for the query. The first runs against the chunker alone and is instant; the second needs an index over the fixture documents.

import pytest
from tests.fixtures.spans import SPANS

@pytest.mark.parametrize("case", SPANS, ids=lambda c: c["query"][:30])
def test_span_survives_chunking(case, docs, chunker):
    chunks = chunker.split(docs[case["doc"]])
    holders = [i for i, c in enumerate(chunks) if case["span"] in c.text]
    assert len(holders) == 1, (
        f"span for {case['query']!r} appears in {len(holders)} chunks; "
        f"a boundary was moved into it" if not holders
        else f"span appears in {len(holders)} chunks (overlap duplicated it)"
    )

@pytest.mark.parametrize("case", SPANS, ids=lambda c: c["query"][:30])
def test_span_chunk_is_retrieved(case, indexed_retriever):
    hits = indexed_retriever.search(case["query"], k=5)
    assert any(case["span"] in h.text for h in hits), (
        f"the chunk containing the answer to {case['query']!r} "
        f"was not in the top 5"
    )

len(holders) == 1 rather than >= 1 is deliberate and catches the opposite bug. With overlap configured, a span near a boundary can appear in two adjacent chunks — which is not wrong for retrieval, but it does mean the same text is indexed twice, competing with itself for a slot in the top k and consuming the context budget twice if both are returned. If your overlap is large enough that duplication is expected, assert >= 1 and add a separate assertion capping how many chunks a span may appear in; do not simply loosen the check and move on.

Run the first test on every commit; it needs no embeddings and takes milliseconds. Run the second whenever the chunker, the embedder or the index configuration changes.

The structures that split badly

• Tables. A row without its header is a list of numbers with no meaning. Character-based splitters cut tables constantly. Either keep tables whole or repeat the header into every chunk containing rows, and assert whichever you chose.
• Numbered procedures. Step 7 separated from “if you are on the enterprise plan” at step 1 produces advice that is exactly wrong for most readers.
• Definition then use. A term defined in one paragraph and used in the next. Split between them and the second chunk is full of pronouns with no referents — and it will still retrieve, because it contains the term.
• Code blocks. Half a function is worse than no function, and a splitter counting characters has no idea a fence was opened.
• Headings. A chunk that begins mid-section has lost the heading that said what it is about. Prepending the heading path to every chunk is cheap and is separately assertable.

The change is not confined to chunking

Two consequences travel with any chunking change and both need checking in the same pull request.

The index must be rebuilt in full. Vectors computed from old boundaries and vectors computed from new ones in the same index produce a retriever that is quietly two systems, and the results depend on which documents happened to be re-ingested. Assert on a count or a build stamp that the whole corpus was reprocessed, and then check that the rebuilt index behaves the way you expect.

And the context budget moves. Larger chunks with k unchanged means more tokens per request, which is a cost and latency change nobody attributes to a chunking commit weeks later. Assert the total characters or tokens for a representative query stay under a ceiling you have written down, so a chunk-size increase that doubles the prompt fails a test rather than an invoice.