OCR With a Vision Model vs a Real OCR Engine

“Which is more accurate” is the wrong first question. A traditional OCR engine and a vision-language model return different data structures, and half the time that difference decides the choice before accuracy is consulted at all.

They do not return the same object

If anything downstream needs to highlight the source region in a viewer, redact a field in the original pixels, or prove to an auditor which part of page 7 a number came from, an engine that emits boxes is not a preference — it is a requirement. Tesseract, and the hosted document services from the major clouds, all emit them. A generalist model asked for coordinates emits an estimate, which is a different thing entirely and is covered on the grounding page.

Conversely, if what you need is “the invoice total as a number”, an OCR engine hands you a bag of words and leaves the entire problem of finding the total to you. That is the trade: the engine is faithful and structurally dumb, the model is structurally clever and unfaithful.

The failure that has no analogue

When an OCR engine cannot read something it produces garbage that looks like garbage: Inv0ice T0ta1, or a low confidence score, or nothing. Every downstream check you would naturally write catches it.

When a language model cannot read something it produces the most plausible continuation, which is fluent, correctly formatted, and wrong. A smudged 1,247.50 comes back as 1,242.50. A part number is normalised to the format the model saw most often in training. A date in an unfamiliar order is helpfully rewritten to ISO — with the day and month swapped. None of these trip a validator, because they are all well-formed.

This is the single most important difference on the page. It does not mean models should not be used for this; it means the check has to move from format validation to cross-validation — against a checksum, an arithmetic identity (do the line items sum to the total?), a second pass, or a real OCR pass.

Confidence, and why you miss it

OCR confidence is not perfectly calibrated, but it is monotone enough to be worth routing on: a page whose mean word confidence collapses is a page to send to a human. There is no equivalent from a chat completion. Logprobs, where the API exposes them, are about token likelihood under the model, and a confidently hallucinated figure has a high one.

The practical substitutes, in descending order of how much they buy you: run the extraction twice at temperature 0 with two different prompt orderings and flag disagreement; ask for the value and a verbatim quote of the surrounding text, then string-match the quote against an OCR pass; give the schema a required not_legible enum value and measure how often it is used.

A cost model you can fill in

The shapes of the two bills are different, which is why the crossover exists at all:

hosted OCR      cost = pages * price_per_page
self-hosted OCR cost = fixed monthly + (pages * ~0)
vision model    cost = pages * (image_tokens * in_rate
                              + out_tokens  * out_rate)

per page, high detail, one page image:
  image_tokens  ~ 1000-1800   (see the pricing page)
  out_tokens    ~ 300-1500    (depends entirely on your schema)

Two consequences. Self-hosted OCR has a fixed cost and a near-zero marginal one, so at high volume it wins on price by an amount that is not close — this is why bulk digitisation projects still run Tesseract. And the model’s output tokens are under your control: asking for the full page as markdown when you needed four fields can be most of the bill.

Latency has a different shape again, and it matters whenever a human is waiting. An OCR engine’s time is roughly proportional to page area and is stable to the millisecond, which makes it easy to put in front of a user. A model’s time is prefill over the image plus generation of every output token, so it scales with how much you asked for — and it inherits the provider’s tail. The p99 of a hosted model call is a different animal from its median, so a synchronous user-facing extraction needs a timeout and a story for what happens when it fires. For an overnight batch none of this matters and you should ignore it.

How to decide, and the hybrid

• Need coordinates, confidences, or reproducibility? OCR engine. This is not a close call and no prompt changes it.
• Millions of pages of similar layout? OCR engine, probably self-hosted, with a model only on the exceptions.
• Messy, varied, low-volume documents where the output is a handful of semantic fields? Model. The layout robustness is genuinely where it earns its keep — no template to maintain.
• Handwriting, or a language whose script your engine handles badly? Test both; see the handwriting page for why this one genuinely varies.

The hybrid is usually the right production answer and is rarely described: run OCR first, then pass both the image and the OCR text to the model in one request. The model gets faithful character evidence to anchor on and pixels for layout and context, and you keep the boxes for provenance. It costs one OCR call plus the image tokens plus a modest text prefix, and it removes most of the silent-correction failure mode because the model is now reconciling two sources rather than guessing from one.

If you go that route, judge candidates on how well they follow a strict output schema rather than on vision leaderboard position — the benchmarks in this space, DocVQA and OCRBench among them, measure question answering over documents, which is a related but not identical job to filling in your fields.