Reordering Extracted Text From a Multi-Column Arabic Document

You extract text from a two-column Arabic report and get lines that are half from one column and half from the other, with the digits inside them reversed. Both problems come from the same assumption in the extractor, and only one of them is about Arabic.

What the broken output looks like

The recognisable symptom is a line that reads as two fragments welded together, where each fragment is grammatical and the join is not. A heading and a body sentence from the opposite column arrive on one line. Paragraphs alternate between two unrelated topics every line or two. Somewhere in it a year appears as 5202 instead of 2025, or a phone number comes out with its digit groups in the wrong sequence.

It is worth separating those before you start, because they have different causes and different fixes. The interleaving is a layout analysis failure and would happen to a two-column English document under some extractors too. The reversed digits are a bidirectional text failure and are specific to a right-to-left script. Fixing the first does not fix the second, and a lot of time gets lost assuming it will.

A PDF has no reading order

The thing to internalise is that a PDF page does not contain paragraphs, columns or sentences. It contains a content stream of drawing operators: a text matrix that sets a position, a font selection, and a show-text operator (Tj or TJ) that paints some glyphs there. The order those operators appear in the stream is the order the generator emitted them, which correlates with reading order only by accident — a typesetter that lays out all body text and then goes back to draw the headings will emit them in that order.

So every text extractor reconstructs reading order by sorting the painted runs geometrically. The default sort is: group runs into lines by their y coordinate, then sort within a line by ascending x. That reconstruction is correct for a single-column left-to-right page and wrong for everything else. A two-column page breaks it because two runs at the same y are on the same visual line but not in the same sentence, and the sort happily joins them.

The exception is a tagged PDF. If the file has a /StructTreeRoot with a real structure tree, the reading order is recorded explicitly and you should use it rather than re-deriving it from coordinates. Check for it first — it costs one line and it makes the rest of this page unnecessary when it is present. Documents exported from accessible workflows and government publications increasingly have one; scanned documents and most designer-produced PDFs do not.

The column failure, precisely

Column detection is normally done with a vertical projection profile: sum the ink in each x position across the page, and look for a sustained valley — a band of x values with no glyphs in it running most of the page height. That valley is the gutter, and it splits the page into column bounding boxes. Then you extract each column independently and concatenate.

For a right-to-left document there is exactly one extra step, and it is the step that gets missed: the columns themselves are ordered right-to-left. The first column of an Arabic report is the one with the largest x coordinates. An extractor that finds the gutter correctly and then concatenates columns in ascending x order produces output that is internally coherent per column but has the document’s two halves swapped, which reads as a much subtler kind of wrong than interleaving and often survives review.

Two things make the gutter harder to find in practice. Justified Arabic text uses kashida elongation to fill lines, so the right and left edges of a column are both hard, which is good; but figures, tables and pull-quotes that span both columns put ink in the gutter for part of the page height, which turns one clean valley into two short ones. The fix is to require the valley only over a majority of the page height rather than all of it, and to treat a full-width element as its own block in the sequence rather than as noise to be sorted into a column.

The second failure: visual order and presentation forms

Now the digits. Unicode stores text in logical order — the order you type and read it — and the Unicode Bidirectional Algorithm, specified in Unicode Annex #9, decides at render time where each run goes on screen. A PDF has already been rendered. Many generators write the glyphs out in visual order, left to right across the page, because that is the order they painted them in.

Extract that naively and you get the Arabic words in reverse sequence and, crucially, the numbers reversed too, because a number embedded in RTL text is itself an LTR run. That is where 5202 comes from. Some producers also write Arabic using the presentation-form blocks — U+FB50–U+FDFF and U+FE70–U+FEFF — which encode each letter’s initial, medial, final and isolated shape as a separate character. Text in those blocks will not match a search for the normal letters in U+0600–U+06FF and needs de-shaping back to base forms.

Detecting this is easy: count how many extracted characters fall in the presentation-form ranges. If it is more than a handful, you are looking at visually-ordered output and every run needs reversing before it is usable. If it is zero but digits still look wrong, the producer emitted logical order and something later in your pipeline reversed it — check any step that concatenates strings with a template.

Recovering the order

1. Check for a structure tree. If /StructTreeRoot exists, take the reading order from it and stop.
2. Extract runs with coordinates rather than as a flat string. Any extractor that gives you a per-word bounding box will do; a plain-text dump has already thrown away the information you need.
3. Build the projection profile over x and find gutters, requiring the valley to hold over at least about 60% of the page height so that a spanning figure does not hide it.
4. Sort column boxes by descending x for an RTL document. This is the one line that fixes the swapped-halves case.
5. Within each column, group runs into lines by y, then sort each line by descending x as well.
6. Normalise: map presentation forms back to base letters with an NFKC pass, and if the text was visually ordered, reverse each run before you store it.
7. Verify on a page you can read: pick a paragraph with a known year in it and confirm the year is not reversed and the sentence does not change topic mid-line.

One warning about the normalisation step. NFKC will also fold other compatibility characters, and on Arabic text it collapses ligatures that you may want to keep distinct if you are doing exact-match search against the original. If that matters, do a targeted mapping of the two presentation-form blocks instead of a blanket NFKC. The trade-off is the same one described in normalising RTL text before indexing, and it is worth deciding once for the whole corpus rather than per document.

Finally, do the reordering before you chunk. A chunker that splits on sentence boundaries will produce nonsense chunks from interleaved text, and because each chunk is individually fluent-looking nothing downstream will flag it — see RAG over RTL PDFs for what that does to retrieval.