Monitoring Document Extraction Accuracy in Production: Audit Samples, Field Metrics and Drift
How to monitor document extraction accuracy in production: a stratified audit sample with independent labels, per-field metrics with intervals, extraction-specific drift triggers, thresholds from your own variance, and regression intake.
- Monitoring document extraction accuracy in production means labelling a small, stratified sample of live documents on a schedule, because schema validity and pass-through checks cannot show whether a value is actually correct.
- Document extraction accuracy should be scored per field, not per document, and each miss classified as missing, wrong or hallucinated, since the three have different fixes (author inference).
- On a small audit sample, report a confidence interval beside every field accuracy, because a point estimate from a few dozen documents can hide a wide range of plausible true rates (general statistical practice).
- Extraction drift usually arrives through a named change, such as a vendor template revision, a parser or model pin change, or a shift in document-family mix, so each of these should trigger a fresh audit sample (author inference).
- Alert thresholds for extraction accuracy should be set from your own baseline labels plus a margin derived from their sample variance, because no universal accuracy figure transfers between document sets.
Monitoring document extraction accuracy in production means periodically labelling a sample of live documents by hand and scoring each field against that label. Nothing else measures field accuracy against ground truth. The Structured Outputs and JSON Mode reference (updated 2026-09-26) describes what constrained decoding gives you, a schema-valid object, and a valid object can still carry a wrong amount. The Document Extraction and Parsing for Agents reference (updated 2026-09-28) separates parsing from field extraction because each has its own failure modes. The AI document extraction pipeline pillar covers the whole pipeline; this piece covers how to know it is still working. Anything not cited to a reference is author inference or general statistical practice, labelled as such.
Build the audit sample by drawing documents from live traffic in strata, then adding a random slice, and have someone label them without seeing the extractor output. The Evaluating AI Agents reference (updated 2026-09-27) says to evaluate your own task distribution because public benchmark scores diverge from production performance; the audit sample is that principle applied to documents. The scheme below is author design.
| Slice | How to draw it | What it catches |
|---|---|---|
| Document family | A quota per family (invoice, contract, form) | A family that quietly degrades while the average holds |
| Vendor or template | A quota per frequent sender or layout | A template revision at one counterparty |
| Risk class | Oversample documents whose fields touch payments, identity or health | Errors with the highest consequence |
| Random slice | A plain random draw across everything | Strata you did not think to define |
Label independently. The labeller should see the page image and the schema, not the extracted record, because a reviewer shown the model's answer can drift toward accepting it (author inference). Where two people label a subset, disagreement shows a field definition that is unclear, which is a schema problem, not a model problem.
The review queue is a biased sample, because documents reach it only after a trigger fires. Its accuracy says nothing about documents that passed unreviewed, and those are the ones monitoring exists to check. How triggers route and how the queue is run is in human review fallback for document extraction; here, use the queue as a source of known failures, never as the accuracy estimate (author inference).
Track accuracy per field, not per document, with exact and normalised match reported separately and every miss classified. A document-level score hides which field broke: one wrong date in twenty fields fails the whole document and tells you nothing about the nineteen right ones (author inference).
| Metric | Definition | Why separate |
|---|---|---|
| Exact match | Extracted string equals the label character for character | Strict; shows formatting drift |
| Normalised match | Equal after a stated normalisation (trimmed, dates to ISO, currency stripped) | Shows value correctness independent of format |
| Missing | Label has a value, extraction is null or absent | Points at parsing or schema description |
| Wrong | Both have values and they differ | Points at reading or reasoning error |
| Hallucinated | Extraction has a value, the label says the field is absent | Points at prompt, enum and null handling |
Write the normalisation rules down and version them, otherwise a rule change looks like an accuracy change. For tables, score rows and cells the way table and form validation for document extraction defines them, and note that a multimodal reader adds its own error types, covered in multimodal document understanding for extraction pipelines.
Report a confidence interval with every field accuracy, because few labelled documents leave wide uncertainty. An illustrative case: 48 correct of 50 labelled values is 96 percent as a point estimate, but a 95 percent Wilson interval runs roughly 86.5 to 98.9 percent. That range is wide enough that a fall to 92 percent next week may be noise. Wilson intervals are standard general practice and the numbers here are an arithmetic illustration, not a measurement of any pipeline. Report each stratum separately; a pooled interval can hide a weak template.
Document pipelines usually drift through a named, checkable change, so list those changes and make each one trigger a fresh audit sample. Generic agent drift, meaning shifting prompts, tools and user behaviour, is covered in online agent metrics and drift monitoring. The triggers below are specific to extraction (author inference).
- Vendor or template change. A counterparty redesigns its invoice. Track the share of documents per template fingerprint (for example a layout hash or sender) and sample any template whose share jumps or that appears for the first time.
- Parser or model pin change. A new parser release or model snapshot changes output without any change on your side. Record both versions on every extraction and re-sample after every bump, as the version fields in the extraction schema versioning approach make possible.
- Shift in document-family mix. Overall accuracy can fall with no per-family change, simply because a harder family became a larger share. Monitor the mix next to the accuracy, and report per-family results.
- Input quality shift. A new scan source or capture app changes resolution or skew. Track simple input stats such as page count and image size alongside results.
Compare each triggered sample against a fixed baseline window, using the baseline discipline in online agent metrics and drift monitoring; re-baseline only after a known, accepted change.
Set each threshold from your own baseline labels, because no source in the corpus publishes an acceptable extraction accuracy and a figure borrowed from another document set does not transfer (author inference). The calibration steps (labelled sample, per-field sort, stricter lines for money fields) are in the pillar's threshold section, and re-calibration after a change is covered in human review fallback for document extraction. This section adds only the alert rule for monitoring.
Use the lower bound of the baseline interval as the alert line, and compare a new sample's interval against it. That rule has a power limit worth stating. With 50 labels per field the baseline lower bound is about 86.5 percent (the 48-of-50 case above), and a new sample's upper bound drops below it only at 38 of 50 correct, or 76 percent. A fall from 96 to 90 percent passes silently (arithmetic illustration, general practice). Two ways to trade this off, both author inference:
- Label more: the interval narrows with sample size, so smaller drops become detectable.
- Two tiers: page only on the strict rule, and open an investigation when the point estimate crosses the line, accepting some false alarms for earlier warning.
Risk shapes the quota too: a field that moves money earns a larger audit sample than a free-text note, so its interval is narrower.
Send disagreements and new templates into the regression set, and record enough to replay them. Regression-set mechanics, meaning how to run and gate it, are in incident-to-eval-fixture-loop; this section covers intake only (author design).
- New tags only: the stratum and the miss type (missing, wrong, hallucinated), because those tie an item back to the audit metrics above.
- Record shape, hashing and schema-version tagging: follow the review-item record in human review fallback for document extraction and the correction migration rules in extraction schema versioning instead of a second format.
- Dedup and caps: skip repeats by content hash, and cap how many items one template contributes so a single noisy sender does not dominate the set.
- Privacy: keep only what your retention rules allow, as the pillar's launch checklist says.
The cost of labelling is a real line item; see cost per page for document extraction for where it belongs in the worksheet.
As of October 2026 ChangeGamer runs no document extraction, so it has no accuracy data to show. The one related practice is build-time contract validation: the loader in src/lib/articles.ts throws at build when an article has a missing or duplicate targetQuery. That is a structural check that catches a class of mistakes cheaply, and it says nothing about whether any article's content is right, which is the same gap between schema validity and accuracy described above.
Frequently asked questions
- How do you measure document extraction accuracy in production without ground truth?
- Production document extraction accuracy is measured by creating ground truth by sampling live documents and having a person label them without seeing the model output. Stratify the sample by document family, vendor template and risk class, and add a random slice to catch strata you did not anticipate. This is general sampling practice and author design, not a rule from the reference corpus.
- How big should a document extraction audit sample be?
- A document extraction audit sample has no universal size; derive it from the precision you need per field. Compute an interval on a pilot sample and keep sampling until the interval is narrow enough to tell a real drop from noise. Rare fields and small strata need their own quota (general statistical practice, author inference).
- What causes document extraction accuracy to drop after launch?
- Accuracy in a document extraction pipeline typically drops because of a vendor changing its template, a parser or model version change, and a shift in which document families arrive. The extraction reference treats parsing and field extraction as separate stages with separate failure modes, so check both stages (author inference on the causes).
- Which production documents should become regression tests?
- Add documents whose audit label disagreed with the extraction, plus one representative per new template or family. Store the page image reference, the corrected record, the parser, model and schema versions, and a content hash so that duplicates are not added twice (author design).
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.