OCR vs Vision Language Model for Document Parsing: How to Choose Stage A
OCR vs vision language model for document parsing: a decision order keyed on residency, layout variance, volume and known schema, how a hybrid arbitrates disagreements, a per-field bake-off, failure-mode mapping, page batching and privacy.
- Choose the Stage A document parser by working through data residency first, then layout variance, then volume, then whether the schema is known, because residency can rule out an option before accuracy matters (the author's inference).
- A hybrid document parser should let OCR win on exact characters such as digits and identifiers and let the vision model win on structure such as which column a value belongs to (the author's inference).
- A parser bake-off should score each field separately across candidates, splitting correct, missing and wrong values, because a page-level score hides one parser failing on the single field that moves money.
- Sending page images to a hosted vision model transfers whole pages, not selected fields, so data minimization is harder than with text; the Data Privacy reference lists provider retention as 30 days at OpenAI (undated in the reference) and 7 days at Anthropic (as of September 2025).
- As of October 2026 ChangeGamer runs no document-extraction pipeline and has no accuracy, throughput or cost figures, so every threshold in this article is a method to measure on your own documents.
Choose the Stage A parser by constraint order: data residency first, layout variance second, volume third and a known schema fourth, then confirm the choice with a per-field bake-off on your own documents. The AI document extraction pipeline pillar explains why parsing and extraction are separate stages; this article is the procedure for picking and running the parser. Where a sentence cites a reference, the claim comes from it; every other rule is the author's inference.
Start with the first constraint that removes an option, and work down this order. The ordering is inference; the facts behind each step come from the Document Extraction and Parsing for Agents reference (updated 2026-09-28).
- Can the documents leave your infrastructure? If not, hosted vision models are out and the choice is among self-hosted OCR, layout tooling or an open-weight vision model. The Multimodal Agents reference (updated 2026-10-04) lists open-weight families such as Qwen3-VL and InternVL3.
- Is layout variance high? Unpredictable layouts favour a vision model or a hybrid, because the extraction reference says vision models handle layout ambiguity that rule-based pipelines miss. Stable templates favour OCR.
- How large is the volume? The reference puts OCR plus layout analysis on high-volume work and vision models on low to medium volume, because vision input costs more per page.
- Is the schema known up front? The reference lists a known schema as a case where a vision model fits. Without one, produce Markdown first and decide later.
If two options survive, test both; the corpus publishes no accuracy ranking for any parser.
A hybrid runs OCR and a vision model on the same page and compares what each returns for every field. The Multimodal Agents reference recommends feeding the OCR text alongside the page image for financial, medical or legal documents, so the model reasons over both with the OCR text as an anchor. The comparison step below is inference.
| Situation | Winner | Route |
|---|---|---|
| Both readers return the same value | Either | Accept, then run validation |
| Values differ in characters (digits, IDs, dates) | OCR text | Accept only if validation passes, else review |
| Vision value does not appear in the OCR text | Neither | Flag as ungrounded and send to review |
| Readers disagree on which cell or column owns a value | Vision model | Verify against the table source |
| OCR returns nothing for a region the vision model reads | Vision model | Review, since the region may be a stamp or handwriting |
OCR should win on characters because the multimodal reference lists hallucinated visual details as a known vision-model failure, naming an invented dollar amount or part number as a real production failure. A character an OCR engine never produced is evidence of that. As inference, the reverse also holds: OCR is weaker at deciding what a number belongs to, which is why layout questions go to the vision model.
Log each disagreement as a reason code and send the field, not the document, to the review path in the pillar. The reviewer needs both candidate values, the page image and the Stage A text around the value.
Run the bake-off by hand-labelling target fields once, running every candidate over the sample, and scoring each field on its own. Page-level scores hide the failure that matters: a parser can read 95 percent of a page perfectly and still miss the total.
- Three outcomes per field. Correct, missing and wrong. A missing field can be routed to review cheaply; a wrong one is silent.
- Slice by document type and hard case. One average across templates and rotated scans tells you nothing about either.
- Weight by consequence. Treat fields that move money or identity as separate pass criteria.
- Cost per accepted field. Managed OCR is billed per page and vision input in tokens; the multimodal reference says token formulas differ by provider and model version. The only per-page prices in the corpus are Mistral's, reported via secondary sources ($4 per 1,000 pages for OCR 4 standard, $2 batch, and $5 for Mistral Document AI, not Google's), so verify vendor pages before budgeting; cost per page for document extraction gives a blank sensitivity table to fill from your pilot.
The script below scores two invented candidates on two invented records; replace the dictionaries with your labelled sample.
python3 - <<'EOF'
from collections import defaultdict
gold = [{"total": "120.50", "iban": "DE89 3704"}, {"total": "88.00", "iban": "FR14 2004"}]
runs = {
"ocr": [{"total": "120.50", "iban": "DE89 3704"}, {"total": "88.00", "iban": None}],
"vlm": [{"total": "120.50", "iban": "DE89 3704"}, {"total": "80.00", "iban": "FR14 2004"}],
}
for name, preds in runs.items():
s = defaultdict(lambda: {"correct": 0, "missing": 0, "wrong": 0})
for g, p in zip(gold, preds):
for field, value in g.items():
got = p[field]
s[field]["correct" if got == value else "missing" if got is None else "wrong"] += 1
print(name, dict(s))
EOF
It prints one line per candidate: OCR leaves one IBAN missing, while the vision model gets one total wrong.
Map each observed failure to the parser change most likely to fix it, then confirm on the bake-off sample. The failure descriptions come from the extraction and multimodal references; the "Seen in" and "Try" columns, and the blank-region row, are inference.
| Observed failure | Seen in | Try |
|---|---|---|
| Wrong reading order on multi-column pages | OCR pipelines | Vision model, or a layout-aware library |
| Rotated or skewed scans misread | OCR pipelines | Deskew first, or a vision model |
| Merged or spanning table cells misaligned | Both | Check against source; hybrid with row-count checks |
| Small or low-contrast text mis-read | Vision models | OCR pass at full resolution, hybrid |
| Invented values | Vision models | OCR anchor plus a grounding check |
| Blank output on a region (inference) | OCR pipelines | Vision model on that region |
Changing the parser fixes only Stage A failures. If values are present in the Stage A text but missing from the record, the cause is the schema or extractor; see structured outputs and JSON mode.
Size the call to the smallest span that contains all the evidence for a field, usually one page, and add the neighbouring page when a table or a total can cross a page break. This is extraction-unit sizing, not retrieval chunking; split strategy for search is in chunking documents for RAG.
- Single-page documents: one call per document.
- Short multi-page documents: one call per page, plus a reconciliation step for fields that can appear on any page. The reconciliation rule is inference.
- Tables across pages: send the last rows of one page with the next page, then deduplicate.
- Hard ceilings: per the multimodal reference, Anthropic accepts up to 100 images per request for 200k-context models and 600 for others, and applies a 2000 px long-edge limit above 20 images.
- Resolution: downscale to the lowest resolution that still reads the smallest text, the reference's main cost lever. Test it on small-print fields, where the reference says vision models fail.
Sending a page image to a hosted model transfers everything printed on the page, so you cannot send only the fields the task needs. The Data Privacy and PII for Agents reference (updated 2026-09-28) says to send only the data a task requires and to redact before the model call; with a raw image there is nothing to redact yet. The compensating pattern, as inference, is to run self-hosted OCR first, redact the text, and send the redacted text or cropped regions.
| Question | Hosted vision model | Self-hosted OCR |
|---|---|---|
| Where do the bytes go? | To the provider | Stay in your infrastructure |
| Retention | OpenAI up to 30 days (undated in the reference), Anthropic 7 days (as of September 2025 per the reference), so re-check before relying on them; OpenAI zero retention needs approval for eligible endpoints | Yours to set |
| Contract | A signed DPA is required under GDPR Article 28, per the reference; take legal advice | No processor for the parse step |
| Operations | Provider's problem | You run and update the models |
For self-hosting, the extraction reference lists Docling (MIT licence) among open-source parsers re-confirmed reachable on 2026-09-28. Check model-weight licences separately: Marker and Surya weights carry revenue thresholds. Mistral reportedly offers a self-hosted OCR 4 container, but that is unverified in the reference, so confirm it with the vendor. This is not legal advice.
As of October 2026 ChangeGamer runs no document-extraction pipeline and has no accuracy, throughput or cost figures, so nothing above is measured on its own documents. One verifiable fact is adjacent: its .md article variants are not produced by a parser. In src/pages/articles/[slug].md.ts the comment states the body is "emitted verbatim from the same source the HTML page renders", and /llms.txt links those .md URLs. The site avoids Stage A by authoring Markdown at the source. Teams that control the origin format can do the same, so parse only the documents you do not author.
Frequently asked questions
- Is OCR or a vision language model better for document parsing?
- No parser wins in general. The Document Extraction reference puts OCR plus layout analysis at high volume and vision models at low to medium volume with complex layouts, and publishes no cross-parser accuracy ranking, so run both on a labelled sample of your own documents and compare field by field.
- What is a hybrid OCR and vision model parser?
- A hybrid parser runs a dedicated OCR pass and gives that text to a vision model alongside the page image. The Multimodal Agents reference recommends this for precision-critical fields, with the OCR text acting as a ground-truth anchor and the image supplying layout context.
- How do you compare document parsers fairly?
- Hand-label the target fields once on a representative sample that includes your worst scans, run every candidate over it, and score each field separately as correct, missing or wrong. Compare per-field results by document type rather than a single page-level accuracy number.
- Is it safe to send document page images to an AI provider?
- Sending page images to a provider is safe only if the data and contract allow it. A page image carries everything printed on the page, so apply data minimization, sign a data processing addendum, and check the provider's retention terms. Where documents cannot leave your infrastructure, parse with self-hosted OCR instead.
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.