Chunking a Bilingual Document Without Splitting Mid-Translation

Contracts, regulatory filings, product manuals and EU documents are routinely published with two languages on one page. Run one through an ordinary ingest pipeline and you get chunks holding the end of a French sentence and the start of its English translation — a unit that is a good match for nothing.

Two layouts, two different bugs

Parallel columns. The source language runs down the left, the target down the right. The bug happens during extraction: an extractor that reads by y-coordinate treats the two columns as one line each, so the text comes out interleaved sentence by sentence, or worse, phrase by phrase. By the time it reaches the chunker the document is already destroyed, and no chunking strategy recovers it.

Alternating paragraphs. A paragraph in one language is followed by its translation. Extraction is correct here — the reading order really is alternating — and the bug is in the chunker, which packs to a size budget and puts a boundary in the middle of a paragraph pair.

Diagnose which one you have before writing any code, because the fixes live at different stages. Extract one page and read the first twenty lines: if the languages alternate within a sentence, it is a column extraction problem; if whole paragraphs alternate cleanly, it is a chunking problem.

Why a mixed chunk ranks badly

An embedding is a single vector summarising everything in the chunk. A chunk that is half French and half English encodes a blend, and because the vector space is organised into language regions, that blend lands between them — closer to neither language’s query region than a clean chunk in either language would be. It loses to the monolingual chunk for a French query and loses again for an English one.

The count makes it worse. A document split into mixed chunks has every chunk in that in-between position, so there is no clean chunk to lose to and the entire document under-retrieves. This is the same mechanism as embedding a mixed-script document, and it is why the fix is structural rather than a matter of tuning the ranker.

There is a narrower harm too. If the chunk holds the second half of a French sentence and the first half of the English one, then neither sentence is complete anywhere in the index, so a query matching the missing half retrieves nothing at all.

Recovering columns before extraction

For a two-column PDF, cluster text spans by their x coordinate before reading anything in order. A two-column layout produces a clear bimodal distribution of left edges, and the gutter between the two modes is the split.

import fitz   # PyMuPDF

def column_split(page, gutter_guess=None):
    """Return two lists of (y, text) — left column and right column."""
    words = page.get_text("words")   # x0, y0, x1, y1, word, block, line, wordno
    xs = sorted(w[0] for w in words)
    if not xs:
        return [], []
    mid = gutter_guess or (page.rect.width / 2)
    left  = [(w[1], w[4]) for w in words if w[2] <= mid]
    right = [(w[1], w[4]) for w in words if w[0] >  mid]
    lines = lambda ws: [t for _, t in sorted(ws, key=lambda p: p[0])]
    return lines(left), lines(right)

def looks_two_column(page, tol=0.08):
    """Cheap check: is there a vertical band with almost no glyphs in it?"""
    words = page.get_text("words")
    width = page.rect.width
    band = [w for w in words if 0.45 * width < w[0] < 0.55 * width]
    return len(band) < tol * max(len(words), 1)

The midpoint is a guess and it is wrong for asymmetric layouts, which are common when one language is systematically longer than the other. Prefer finding the actual gutter: histogram the x0 values into bins, look for the widest empty run in the middle third of the page, and split there. Run looks_two_column per page rather than per document, because front matter, tables and appendices are frequently single-column in an otherwise two-column file.

Segmenting into single-language runs

Once the text is in correct reading order, the chunker’s job is to never let a boundary between languages fall inside a chunk.

1. Split the document into paragraphs — blank lines, or PyMuPDF blocks, or whatever structure your source gives you. Paragraphs are the granularity at which language changes.
2. Run language identification on each paragraph. Use a detector that returns a confidence, and treat anything under your threshold as “unknown” rather than guessing.
3. Merge consecutive paragraphs sharing a language into a run. Assign an “unknown” paragraph — a table, a code listing, a heading of proper nouns — to the run it sits inside rather than starting a new one.
4. Chunk within each run independently, with the token budget and segmentation rules appropriate to that run’s language.
5. Tag each chunk with its language and with an identifier for the source section, so the two languages’ chunks can be joined later.

def language_runs(paragraphs, detect, min_conf=0.7):
    runs, cur, cur_lang = [], [], None
    for p in paragraphs:
        lang, conf = detect(p)
        if conf < min_conf:
            lang = cur_lang            # inherit: tables, headings, numerals
        if cur and lang != cur_lang:
            runs.append((cur_lang, cur)); cur = []
        cur.append(p); cur_lang = lang
    if cur:
        runs.append((cur_lang, cur))
    return runs

def chunk_bilingual(paragraphs, detect, chunkers, section_id):
    out = []
    for i, (lang, paras) in enumerate(language_runs(paragraphs, detect)):
        chunker = chunkers.get(lang, chunkers["default"])
        for c in chunker("\n\n".join(paras)):
            out.append({"text": c, "lang": lang,
                        "pair_id": f"{section_id}:{i // 2}"})
    return out

The pair_id is the piece that turns this from a segmentation fix into something useful. Successive runs in an alternating document are translations of each other, so integer-dividing the run index by two groups a run with its counterpart. That is a heuristic and it breaks the moment a run is missing, so where the document has section numbering, use that instead — the numbers are usually identical across the two languages, which is a far more reliable join key.

Indexing the pair

With clean runs and a pair identifier you have three usable policies.

• Index both, return both. Every chunk is embedded in its own language, and when one is retrieved its counterpart is fetched by pair_id and attached. Best recall — a query in either language finds the section — at the cost of doubling the index. This is usually the right default for legal and regulatory corpora, where the reader wants both texts anyway.
• Index one, keep the other as payload. Embed only the canonical language and store the translation as an unindexed field. Half the vectors; queries in the non-canonical language rely on the embedding model’s cross-lingual alignment, which is the tradeoff described in retrieval when query and document are in different languages.
• Index both, deduplicate at retrieval. Both are embedded, but the result set collapses any two hits sharing a pair_id into one, keeping the higher-scoring text. Avoids the top-k filling with two copies of the same passage, which is the usual complaint about the first policy.

Whichever you pick, add one assertion at ingest: no chunk may contain text from more than one run. It is the invariant this entire page exists to establish, it is one line to check given the run structure, and it is the thing that silently stops holding when someone adds a new document source.