Reading On-Screen Text in Video Frames

A scanned page is flat, sharp, high-contrast, axis-aligned and photographed once. A sign in a video frame is none of those things, and you get sixty of them a second. Both halves of that sentence change the engineering.

What makes a frame harder than a page

The literature draws a firm line between document OCR and scene text recognition, and the competitions that define the field draw it too: the “incidental scene text” task in the ICDAR Robust Reading Competitions, hosted by the Computer Vision Center at the Universitat Autònoma de Barcelona at rrc.cvc.uab.es, exists precisely because methods that worked on documents did not transfer. Video adds a further set of degradations on top of scene text.

• Perspective and curvature. Text on a sign photographed at an angle is a projective transform of a rectangle; text on a bottle or an arch is genuinely curved. A recogniser expecting a horizontal strip gets neither.
• Motion blur. A moving camera or a moving subject smears character strokes along the direction of motion during the exposure. This is the video-specific degradation and it is destructive in a way that low resolution is not: blur removes the high-frequency detail that distinguishes similar glyphs, and no amount of upscaling recovers it from one frame.
• Rolling shutter. On CMOS sensors the frame is read out progressively, so fast horizontal motion skews vertical strokes. Straight edges lean, which upsets both detection boxes and rectification.
• Compression artefacts. Text is high-contrast high-frequency content, which is what block-based codecs handle worst. Ringing around glyph edges and blocking in flat areas around them are introduced by the encoder, are worse at low bitrate, and are not present in the original at all. A crop of a caption from a low-bitrate stream can be markedly harder to read than the same caption from the master.
• Interlacing and telecine. Older or broadcast-derived material can present combed edges on moving text unless deinterlaced first, and a deinterlacer choosing the wrong field halves the vertical resolution of the text.

Detect, rectify, recognise

The conventional pipeline has three stages and it is worth keeping them separate, because they fail differently and you will want to know which one is failing.

1. Detection finds where text is. The influential approach here is CRAFT, described by Baek and colleagues in their 2019 paper, which predicts a per-character region score and an affinity score between adjacent characters rather than predicting word boxes directly. Because words are assembled from character-level evidence, it handles long, curved and arbitrarily shaped text that a rigid word-box detector cannot represent at all. Other families predict segmentation maps or rotated boxes; the choice matters most for curved text.
2. Rectification warps the detected region to a horizontal strip of fixed height. For a quadrilateral this is a perspective transform; for curved text it needs a thin-plate spline or a learned rectifier. Skipping this stage is a common cause of a recogniser that works on test images and fails on real footage.
3. Recognition turns the strip into characters. The long-standing baseline is a convolutional backbone feeding a recurrent layer trained with connectionist temporal classification, which handles variable-length output without needing per-character alignment. Attention-based decoders and models with an explicit language-model component do better on degraded input because they can use linguistic context to resolve an ambiguous glyph — which is also how they introduce their characteristic failure of hallucinating a plausible word that is not there.

Sixty chances at the same string

This is the part that has no analogue in document OCR and it is where most of the available accuracy lives. A sign visible for two seconds at 30 fps appears in sixty frames. Those are sixty different observations of the same underlying string, with different blur, different sub-pixel alignment, different compression noise and different lighting. Treating them as sixty independent OCR jobs and returning sixty slightly different strings is the naive implementation, and it is worse than useless downstream.

The correct structure is to track first and read afterwards.

• Associate detections across frames. A text region in frame t and one in frame t+1 that overlap heavily, after compensating for camera motion, are the same physical text. This is ordinary object tracking applied to text boxes, and it turns a pile of detections into tracklets.
• Vote across the tracklet. Run recognition on every frame of the tracklet and combine per character position, weighting each frame by the recogniser’s own confidence and by a sharpness measure. A character misread in twelve frames and read correctly in forty-eight resolves correctly; the same character read once resolves however that one frame happened to look.
• Prefer sharp frames. Rather than voting over all sixty, select the least-blurred few — the variance of the Laplacian over the crop is the standard cheap sharpness proxy — and recognise only those. This is usually the better trade: it cuts recognition cost by an order of magnitude and improves accuracy, because blurred frames contribute noise to the vote rather than information.
• Emit once, with a time span. The output of the tracklet is one string with a start and end timestamp, not sixty rows. Everything downstream — indexing, moderation, subtitle extraction — wants that shape.

There is a further gain available from the same redundancy. Because consecutive frames sample the scene at slightly different sub-pixel offsets, several low-resolution observations of static text carry more information jointly than any one of them does alone, which is the basis of multi-frame super-resolution. It is worth reaching for only when the text is genuinely too small in every individual frame, and it requires accurate sub-pixel registration to help rather than smear.

The height threshold that decides everything

One number predicts success better than any architectural choice: the height of the text in pixels in the source frame. Recognisers are trained on strips of a fixed height, commonly 32 or 48 pixels, and text that arrives smaller than that has to be upscaled — which invents no detail and often amplifies compression artefacts.

The practical consequence is that a caption occupying 3% of the height of a 1080p frame is about 32 pixels tall and readable; the same caption in a 360p re-encode is about 11 pixels tall and generally is not. This is the single strongest argument for running text recognition on the highest-resolution copy you have rather than on the proxy you use for everything else, even though the proxy is cheaper to decode. Check the documented input height of the specific recogniser you are using and compare it against the actual pixel height of the text you care about before concluding that the model is bad.

Not all on-screen text is the same job

A pipeline that returns one undifferentiated stream of strings has discarded information the downstream consumer needs. Three categories behave differently enough to be worth separating at the point of detection.

• Burned-in captions and subtitles. Static position, consistent styling, high contrast, and semantically a transcript. If a real subtitle track exists, extract that instead — it is exact, free and timed. Where it does not, recognised captions are the fallback and they duplicate the audio transcript, which means de-duplicating against it rather than indexing both.
• Graphics and lower thirds. Names, scores, headlines, slide content. Usually the highest-value text in the frame for search and usually the easiest to read. Position is a strong prior and worth exploiting.
• Incidental scene text. Signs, packaging, screens within the shot, licence plates. The hardest case, the one the ICDAR incidental task is built around, and the one with the clearest privacy implications — reading every visible screen and document in footage is a data collection decision, not just a technical one.

For moderation specifically, on-screen text is the channel that neither the audio transcript nor a general image classifier covers, which is why it appears as its own stage in the moderation pipeline. General single-image OCR concerns — language coverage, script direction, model selection — are the subject of the OCR pipeline page and apply unchanged once the frame has been extracted and rectified.