OCR Pipelines: The Preprocessing That Decides Your Accuracy

OCR accuracy is decided mostly before the OCR engine runs. Tesseract documents this itself: its “Improving the quality of the output” page is a list of image problems, not engine settings, and the order it lists them in is close to the order of their impact.

Resolution first, everything else second

Tesseract’s documentation asks for at least 300 DPI, and notes that it is the height of the characters in pixels that actually matters — roughly 30 pixels of x-height is where accuracy stops improving. This is the one input you often cannot fix afterwards: a 150 DPI scan upscaled to 300 DPI has no more information in it than it had, and interpolation invents edges that the binariser then commits to.

Where you can control it — rendering a PDF page to an image for OCR — render at the resolution you want rather than rendering at the default and resizing:

import fitz          # PyMuPDF

def render(page, dpi=300):
    # PDF user space is 72 units per inch, so the zoom factor is dpi/72.
    m = fitz.Matrix(dpi / 72, dpi / 72)
    return page.get_pixmap(matrix=m, colorspace=fitz.csGRAY)

Rendering greyscale rather than colour halves the memory and loses nothing the binariser was going to keep. For a page that is already an image inside the PDF, extracting the embedded image directly is better still — rendering re-samples it, and re-sampling before binarisation is exactly the thing you are trying to avoid.

Binarisation, and when Otsu is wrong

Tesseract binarises internally — via Leptonica, using Otsu’s method by default — so the question is not whether to binarise but whether the default will do it well. Otsu picks a single global threshold by maximising between-class variance, which is the right model when the page has one background level.

It is the wrong model for a photograph of a page, where one side is brighter than the other, and for a scan with a shadow along the spine. A global threshold on those turns the dark side into a solid black block and takes the text with it. The fix is a local threshold — Sauvola or Niblack, or OpenCV’s adaptive threshold, which computes a threshold per neighbourhood:

import cv2, numpy as np

def binarise(gray: np.ndarray, uneven: bool) -> np.ndarray:
    if not uneven:
        _, out = cv2.threshold(gray, 0, 255,
                               cv2.THRESH_BINARY + cv2.THRESH_OTSU)
        return out
    # blockSize must be odd and larger than a character; C is subtracted
    # from the local mean, so raising it thins strokes.
    return cv2.adaptiveThreshold(gray, 255,
                                 cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
                                 cv2.THRESH_BINARY, blockSize=31, C=10)

def looks_uneven(gray: np.ndarray, tiles: int = 4) -> bool:
    h, w = gray.shape
    means = [gray[y:y+h//tiles, x:x+w//tiles].mean()
             for y in range(0, h - h//tiles + 1, h//tiles)
             for x in range(0, w - w//tiles + 1, w//tiles)]
    return (max(means) - min(means)) > 40      # tune on your own scans

Note the shape of that: a detector that decides which treatment a page needs, rather than one treatment applied to everything. Applying adaptive thresholding to a clean 300 DPI scan makes it slightly worse, which is why “always preprocess” is bad advice and “preprocess conditionally” is not.

Deskew and despeckle

Tesseract’s line-finding tolerates a small rotation and stops coping somewhere past a couple of degrees, which a sheet-fed scanner or a phone camera will exceed easily. The classical fix is to threshold, find the minimum-area rectangle around the ink, and rotate by its angle:

def deskew(gray: np.ndarray) -> np.ndarray:
    inv = cv2.bitwise_not(gray)
    _, bw = cv2.threshold(inv, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
    coords = np.column_stack(np.where(bw > 0))
    angle = cv2.minAreaRect(coords)[-1]
    if angle > 45:          # OpenCV reports (0, 90]; normalise to (-45, 45]
        angle -= 90
    if abs(angle) < 0.3:    # don't resample for nothing
        return gray
    h, w = gray.shape
    m = cv2.getRotationMatrix2D((w / 2, h / 2), angle, 1.0)
    return cv2.warpAffine(gray, m, (w, h),
                          flags=cv2.INTER_CUBIC,
                          borderMode=cv2.BORDER_REPLICATE,
                          borderValue=255)

The abs(angle) < 0.3 guard matters: every rotation resamples the image and softens the glyph edges, so rotating a page that was already straight is a small loss with no gain. Despeckling — removing connected components below a few pixels — helps on photocopies and faxes and does nothing on clean scans, so gate it the same way.

Page segmentation is a parameter, not a constant

Tesseract’s --psm flag chooses how it looks for text layout, and the default (3, fully automatic page segmentation without orientation detection) is wrong for a lot of what you will feed it. --psm 6 assumes a single uniform block of text and is consistently better on cropped regions; --psm 7 treats the image as one line, which is what you want for a cropped field on a form; --psm 11 is sparse text with no ordering, for labels scattered over a diagram. Passing a whole receipt at --psm 3 and a cropped total field at --psm 7 are different jobs on the same document.

Two other flags earn their place. -l takes a +-joined list of languages and the traineddata you name must be installed. And -c tessedit_char_whitelist=0123456789.,- restricts the output alphabet, which on a numeric field removes the entire class of digit-to -letter confusions (0/O, 1/l, 5/S, 8/B) by construction rather than by post-processing.

Measuring what each step is worth

Whether deskewing helps your documents is not something a page can tell you; it depends on how your documents were digitised. It is also cheap to find out. Transcribe thirty representative pages by hand, then run the ablation:

import itertools, jiwer, pytesseract

STEPS = {"deskew": deskew, "binarise": lambda g: binarise(g, looks_uneven(g))}

def ablate(pages):        # pages: list of (gray_image, ground_truth_text)
    for r in range(len(STEPS) + 1):
        for combo in itertools.combinations(STEPS, r):
            errs = []
            for img, truth in pages:
                out = img
                for name in combo:
                    out = STEPS[name](out)
                got = pytesseract.image_to_string(out, config="--psm 6")
                errs.append(jiwer.cer(truth.lower().split(),
                                      got.lower().split()))
            print(f"{'+'.join(combo) or 'raw':20} CER {sum(errs)/len(errs):.4f}")

Character error rate rather than word error rate, for the same reason as in the PDF parser bake-off: a single wrong character inside a long word costs one CER unit and a full word error, which over-weights long words. Run the ablation once per document class — a class where deskewing is worth a lot and a class where it does nothing will average into a misleading middle if you pool them.

Using the confidence output

Plain image_to_string throws away the most useful thing Tesseract produces. The TSV output gives one row per recognised word with a conf column from 0 to 100 and a bounding box, and -1 for structural rows that are not words:

data = pytesseract.image_to_data(img, config="--psm 6",
                                 output_type=pytesseract.Output.DICT)
words = [(t, c) for t, c in zip(data["text"], data["conf"])
         if t.strip() and int(c) >= 0]
mean_conf = sum(int(c) for _, c in words) / max(len(words), 1)

The bounding boxes are worth keeping too, not only the text. They are what lets a citation highlight a region of the scanned page rather than quoting text the user cannot find, and for a form or an invoice they are how you associate a value with the label to its left. Both of those become impossible the moment you flatten the TSV to a string, and neither can be recovered without running OCR again.

Store mean_conf per page. It is the routing signal for the whole pipeline: pages above your threshold go straight to the index, pages below it go to a second pass — a different preprocessing profile, or a vision model, or a human. It is also the field that lets you answer “how much of this corpus do we actually trust?” without re-running anything.