ChangeGamer

Multimodal Document Understanding for Extraction: Charts, Diagrams, Photos, Stamps and Video Frames

Part 9 of Document extraction pipelines in production · 1,443 words · ~7 min read · published · updated · Markdown variant

How to handle non-text visuals in a document extraction pipeline: when OCR is not enough, extracting a value versus a description versus a class, region-level routing, figure-aware page packing, poor scans, video frames, image-borne injection and per-visual evaluation.

In short

  • Text and table OCR is not enough once a document carries meaning in charts, diagrams, photos, stamps, signatures or handwriting, because a plain transcription of the page drops or garbles exactly those regions (author inference).
  • Before extracting from a visual, decide whether the pipeline needs a value, a description or a classification, because each has a different failure mode and a different check; a value read off a chart is an inference, not a transcription (author design).
  • Image-derived facts should be treated as unverified until cross-checked against page text, a table or a second pass, because vision-language models can describe objects or text that are not in the image (Multimodal Agents resource, updated 2026-10-04).
  • Text rendered inside an image can carry adversarial prompt-injection instructions in the same way web-page text can, so every image region needs the same untrusted-input handling as text from the web (Multimodal Agents resource, updated 2026-10-04).
  • Each visual type needs its own labelled sample and its own error rate, because a single accuracy figure for a whole document type hides that stamps, charts and handwriting fail differently (author design).

Part of the AI Document Extraction in Production: Parse, Extract, Validate, Review guide.


Text and table OCR stops being enough the moment a document carries meaning in something that is not text: a chart, a diagram, a photo, a stamp, a signature, handwriting or a video frame. This article covers those non-text visuals inside the AI document extraction pipeline. It does not repeat how to choose between OCR, vision and hybrid parsing, which the sibling article on that choice settles. Facts come from Multimodal Agents (updated 2026-10-04) and Document Extraction and Parsing for Agents (2026-09-28); everything else is author design or inference, labelled as such.

When is text and table OCR not enough?

Text and table OCR is not enough when the information you need lives in pixels that are not characters: plotted data, drawn relationships, physical objects, marks and signatures. The Document Extraction resource lists overlapping text and image regions among the hard cases for traditional OCR pipelines, and an OCR pass over a chart yields axis labels but not the plotted values (author inference).

Typical plain-OCR output below is author inference, not a measured result.

Visual What plain OCR returns What the pipeline may need
Bar or line chart Axis labels, legend text Plotted values, trend
Diagram or flowchart Box labels in arbitrary order Which node connects to which
Photo Nothing, or stray text on a label A description or a class (damaged, intact)
Stamp or seal Fragments of ring text Presence, type, legibility
Signature Nothing useful Present or absent, whether it sits in the right place
Handwriting Often garbled Transcription with a confidence flag

Dense tables are a separate problem with their own checks, covered in table and form validation.

What should you extract from a visual: a value, a description or a class?

Decide first whether you need a value, a description or a classification, because the three have different failure modes and the choice sets the check. This framing is author design.

Columns below are author design.

Target Example Typical failure Check
Value A bar height as a number Wrong number that looks plausible Cross-check against page text or a table
Classification Stamp present or absent Missed or invented mark Labelled sample per class
Description A caption for a photo Detail that is not in the image Human spot-check; never feed it to a ledger

How far can you trust a number read off a chart?

A number read off a chart is an inference from the plotted geometry, not a transcription, so it should never be the only source for a committed figure. Multimodal Agents lists extracting values from bar charts as a strength and, in the same page, warns that models sometimes describe objects or text that are not present and that image-derived facts stay unverified until cross-checked. Hold both statements: use the chart value as a candidate, compare it with any table or sentence in the document that states the same figure, and send mismatches to review. The resource also advises specific prompts such as listing every item visible, rather than "describe this image", because specificity reduces hallucinated detail.

How do you handle figure regions separately from text?

Crop figure regions and route them to a vision step, while text and tables stay on the cheaper OCR path. Per the resource, the Unstructured library partitions documents into typed elements including Image, which gives you a region type to route on. The routing rule itself is author design.

  1. Detect regions and their types in the parse (Stage A).
  2. Send text and table regions down the OCR path.
  3. Crop each figure region and send it with a narrow, typed prompt and a schema for the answer.
  4. Store the crop, the region coordinates and the page number as provenance.

Some open-weight models can return coordinates. Per the resource, Molmo 2 and Qwen3-VL support returning spatial coordinates for grounding, so a model can say where a stamp or signature sits. The resource makes no accuracy claim and publishes no benchmark ranking, so treat coordinates as a feature to test on your own pages, not as a guarantee (author inference).

How should pages be packed so figures keep their context?

Keep each figure together with its caption and the sentence that refers to it, because a cropped chart without its title and units cannot be read correctly. This is author design. Three habits help:

For batching, page ceilings and how to size requests, follow the OCR versus vision versus hybrid article rather than this one. If the same figure content also feeds retrieval, the chunking decision belongs to chunking documents for RAG.

What goes wrong with scanned photos and low-quality images?

Small or low-contrast text is the documented failure mode, and scans, faxes and phone photos produce a lot of it. Multimodal Agents lists dense fine print, watermarks and text on complex backgrounds as frequently misread, and its remedy is to combine a dedicated OCR pass with the vision model for precision-critical fields.

In practice (author design):

Image resolution and what it costs are out of scope here; see cost per page.

Can an extraction pipeline use video frames?

Often yes: the Multimodal Agents resource states that most major provider models accept video frames or short clips, and tells you to check current documentation for frame-count and duration limits. This article quotes no limit because none is in the corpus and limits change.

For video such as a recorded inspection, how you choose frames is a design decision, labelled as author design: sample on a schedule, on detected change, or around a trigger event, and keep the timestamp of every frame you use as provenance. Treat each extracted statement as an image-derived fact needing a cross-check, and expect counting or spatial questions to be unreliable, since the resource lists spatial reasoning and counting as weak without explicit grounding support.

Can text inside an image attack the pipeline?

Yes: text inside an image can carry adversarial instructions, and the model reads it as part of its context. The resource says this works the same way as text on a web page and recommends the same untrusted-input treatment. The Document Extraction resource adds that OCR will transcribe instruction-style wording hidden in a booby-trapped PDF, so embedded commands should be stripped or escaped before they reach a context.

The image-specific rule is to treat every image region as untrusted input and to flag a visual that "asks" for something. Permissions, output validation and the wider design are covered in prompt injection defense in depth for agents and in the pillar's security section.

How do you evaluate each visual type?

Evaluate each visual type on its own labelled sample with its own error rate, because one accuracy figure hides that stamps, charts and handwriting fail differently. This is author design.

  1. Collect real examples per visual type, including bad scans.
  2. Label the correct answer in the form you extract: value, class or reviewed description.
  3. Report error rate per type and per target kind, not per document.
  4. Set a review rule per type; chart values and handwriting usually start at full review.

Calibration, queues and sampling are in human review fallback for document extraction. Run in shadow beside human keying before any visual-derived field goes live.

What does ChangeGamer know first-hand about this?

As of October 2026 ChangeGamer runs no document extraction, so it has no first-hand error rates, chart-accuracy figures or video results to report. Its multimodal-agents resource was last updated on 2026-10-04 and is a researched reference, not a log of runs. Treat every threshold and routing rule here as a hypothesis to test on your own labelled documents.

Frequently asked questions

Can a vision-language model read the values in a chart reliably?
A vision-language model can extract values from bar charts, but the numbers are inferred from the plotted geometry and are not transcribed, so treat them as unverified. Cross-check against any data table or page text that states the same figures, and route disagreements to a person (author design, based on the Multimodal Agents resource, 2026-10-04).
Should I send every page of a document to a vision model?
No. Send only the pages or regions that contain non-text content and keep text and tables on the cheaper OCR path. The split is author design; the base trade-off between OCR, vision and hybrid parsing is covered in the sibling article on that choice.
Can an extraction pipeline process video frames?
Often yes, because most major provider models accept video frames or short clips, per the Multimodal Agents resource (2026-10-04). Frame-count and duration limits change, so check the current documentation of your provider before designing around any number.
Can text inside a scanned image attack my extraction pipeline?
Yes. Instruction-style text embedded in an image or scan is read by the model as part of its context and may override constraints, so strip or escape embedded commands (Multimodal Agents and Document Extraction resources). Giving the extraction call no write permissions or tools is the author's design choice, not a claim from those resources.
Does ChangeGamer run a multimodal document extraction pipeline?
No. As of October 2026 ChangeGamer runs no document extraction, so this article is design guidance and inference built on its reference corpus, not measured results from a production pipeline of its own.

#document-extraction #multimodal #vision #charts #production

Put this corpus inside your own agents

This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.

Agents: this page as Markdown · JSON · offers at /api/pricing.json · payment methods at /api/payment.json · single-resource access via HTTP 402 (how that works)