Handwriting and Low-Quality Scans

Printed text is a solved problem with a hundred glyph shapes per font. Handwriting has as many shapes as there are writers, no reliable character segmentation, and a legibility that depends on whether the writer was in a hurry. It is the case where general models and specialised engines have genuinely different strengths, and the choice is not obvious in either direction.

Why handwriting is a different problem

• No segmentation. Cursive letters connect. There is no whitespace to cut on, so the classical segment-then-classify pipeline that works on print does not apply, which is why handwriting recognition went sequence-to-sequence long before print OCR needed to.
• Unbounded variation. Slant, size, baseline drift, pressure, and letterforms that are genuinely ambiguous in isolation. A standalone u and v, or n and r, may be indistinguishable without the word around them.
• Historical hands are foreign scripts. Secretary hand, Kurrent, Sütterlin — nineteenth-century German Kurrent is unreadable to a modern German speaker without training, and models trained predominantly on modern text have seen very little of it.
• The medium fights you. Bleed-through from the reverse of a page, foxing, faded ink, ruled lines crossing descenders. Preprocessing matters more here than anywhere else in this cluster.

Context: the advantage and the trap

A general vision-language model brings a strong language prior to the task, and on ordinary modern handwriting that is exactly what is needed. Where a per-character classifier sees an ambiguous squiggle, a model that has read a great deal of English knows that after “Dear Mr” a surname follows, and that “rec_ipt” is “receipt”. Filling that in is not cheating; it is what human readers do.

The trap is that the same prior operates when it should not. On an archival document it will modernise spelling, regularise a non-standard name, silently expand an abbreviation, and normalise an unfamiliar date format. For a search index over correspondence, that is harmless or even helpful. For a transcription that a historian will cite, it is a fabricated source, and the damage is invisible because the output is clean and confident.

So the question is not “which is more accurate” but “is a fluent plausible reading acceptable, or do you need a faithful one, uncertainty included”. Those two requirements point at different tools, and no amount of prompting turns one into the other.

Measuring it: CER, worked by hand

Word error rate is too coarse for handwriting, because one wrong character makes a whole word wrong. The standard metric is character error rate: Levenshtein distance between reference and hypothesis, divided by the reference length.

reference   Received of Mr Smith the sum of five pounds
hypothesis  Received of Mr Smyth the sum of fine pounds

char 1:  i -> y   substitution
char 2:  v -> n   substitution

CER = (S + D + I) / N = (2 + 0 + 0) / 42 = 0.048  ->  4.8 %

WER on the same pair = 2 / 9 = 22 %

Two things to note. The same transcription looks four times worse under WER, so never compare a CER figure from one source with a WER figure from another. And a 4.8 % CER here included changing a surname, which for a genealogical index is a total failure of the only field that mattered — which is the argument for tracking entity accuracy on names, dates and amounts alongside the aggregate.

# jiwer computes both, and normalisation choices dominate results
import jiwer
t = jiwer.Compose([jiwer.ToLowerCase(),
                   jiwer.RemovePunctuation(),
                   jiwer.RemoveMultipleSpaces(), jiwer.Strip()])
print(jiwer.cer(ref, hyp, truth_transform=t, hypothesis_transform=t))

State your normalisation when you report a number. Whether case and punctuation are stripped can move a CER figure by several points, and it is the most common reason two people measuring the same system disagree.

What the specialists still do better

Dedicated handwritten text recognition — the Transkribus ecosystem, Kraken, and the academic line of work benchmarked on the IAM handwriting database — retains several structural advantages worth knowing about before dismissing them as legacy.

• They can be trained on one hand. Give a system a few dozen transcribed pages from a single writer and it specialises to that writer’s letterforms. For an archive of one person’s correspondence, this is decisive and no general model matches it.
• They emit line and word coordinates. Which is what makes an aligned reading view possible, where a scholar clicks a word and sees the ink.
• They emit alternatives and confidences. An uncertain reading can be flagged for human review rather than silently committed.
• They do not paraphrase. Whatever their errors, they are transcription errors, not editorial ones.

What to do with a box of scans

Preprocess first — it is cheaper than any model decision and often worth more. Deskew, convert to greyscale, apply local adaptive thresholding rather than a global one (bleed-through and uneven lighting defeat a global threshold), and crop to the text block. Scan at 300 dpi or better and keep the original: you can always downscale later, and you can never recover what you did not capture.

Then transcribe a stratified sample of thirty pages by hand — clean, typical and awful — and run both candidate systems against it with the CER method above. That afternoon of work is the only thing that answers the question for your box of scans, and the answer genuinely differs between one archive and the next.

If you do use a general model, the prompt does real work here and it is not the usual kind. Ask explicitly for a verbatim transcription including original spelling, abbreviations and line breaks, state that modernising is not wanted, and require an explicit marker — [illegible] is the archival convention — for anything that cannot be read. That last instruction is the important one: without somewhere to put uncertainty, the model has no option but to guess, and the guess is indistinguishable from a reading. Asking for the marker converts an invisible error into a visible gap, and a page with six bracketed gaps is far more useful to a researcher than a page that silently invented six words.

Two more things worth doing on any serious transcription job. Process one line or one region at a time rather than a whole page, because the crop is larger relative to the encoder’s input and the model has less opportunity to drift; segmentation into lines is exactly what the classical tools are good at, so this is another place the hybrid wins. And transcribe each page twice, then diff the two outputs — the disagreements land almost precisely on the words that were hard to read, which gives you a free review queue sorted by difficulty.