# Writing Extracted Document Data to a System of Record: Matching, Keys, States and Replay

> How to write extracted document data to a system of record: matching an extracted record to an existing one, retry keys versus business keys, a staged-to-committed state machine, per-record versus per-document atomicity, dead-letter queues, replay from the saved parse artifact and the write-boundary log.

Guide: Document extraction pipelines in production — part 8
Published: 2026-10-28 · Updated: 2026-10-28 · 1390 words · ~1849 tokens (estimate)
Canonical: https://changegamer.ai/articles/feeding-extraction-output-into-a-system-of-record
JSON: https://changegamer.ai/api/articles/feeding-extraction-output-into-a-system-of-record.json
Access: free — editorial guide, not part of the licensed corpus
Pillar: https://changegamer.ai/articles/ai-document-extraction-pipeline-in-production.md

## In short

- A file hash identifies one upload and makes retries safe, but a re-scan of the same paper document produces different bytes and a different hash, so duplicate detection also needs a business key built from fields such as supplier, document number and date (author design).
- Matching an extracted record to an existing record works best as a ladder: exact business-key lookup first, fuzzy comparison second, and a human decision for anything the first two cannot settle with confidence (author design).
- A staged-to-committed state machine for extracted records should let only code with its own write permission, or a named human reviewer, promote a record, and never the extraction call itself (author design).
- Saving the parse artifact for every document lets a failed or rejected write be replayed from the stored parse without paying for parsing again, and a dead-letter queue holds the records that exhausted their retries (author design).
- Whether a document commits atomically as a whole or record by record depends on whether its records are meaningful alone; an invoice header and its lines usually are not, so they commit together (author inference).

---

Write extracted document data to a system of record through code that is separate from the model: match each extracted record to an existing one, stage it, validate it, and promote it under a rule about who may. The [AI document extraction pipeline](/articles/ai-document-extraction-pipeline-in-production) pillar already covers the basics in [its system-of-record section](/articles/ai-document-extraction-pipeline-in-production#how-should-extracted-data-reach-the-system-of-record): staging, an idempotency key, provenance, and no direct writes. This piece does not repeat them. It covers the decisions that follow once the basics are in place. The extraction call itself returns a record (author inference; see [Structured Outputs and JSON Mode](/resources/structured-outputs-and-json-mode) for the output-shape mechanics), and [Document Extraction and Parsing for Agents](/resources/document-extraction-for-agents) (2026-09-28) describes the parsing stage that precedes it. Everything on the write side is author design or inference, labelled as such. Nothing here is legal advice.

## How do you match an extracted record to an existing one?

Match in a ladder of three rungs: an exact business-key lookup, then a fuzzy comparison, then a human decision for whatever remains. This ladder is author design, not a published standard, and it exists because each rung is cheaper and safer than the next.

| Rung | Method | Outcome | Risk if misused |
|---|---|---|---|
| 1. Exact | Look up the business key (for example supplier, document number, date) | Same record found, or none | Low; a normalisation gap makes a miss look like a new record |
| 2. Fuzzy | Compare normalised names, amounts and dates within a tolerance | Candidate matches with a score | A wrong merge overwrites a correct record |
| 3. Human | A reviewer sees the extracted record beside the candidates | A decision with a named approver | Slow, so reserve it for real ambiguity |

Normalise before every comparison: trim whitespace, fold case, strip punctuation from identifiers and parse dates into one format. Most exact-match misses on identical documents come from formatting differences, not from a different document (author inference).

### When may a fuzzy match act on its own?

A fuzzy match should never merge records on its own; it may only propose candidates for a person or a stricter rule to confirm. Treat the score threshold as something you calibrate on labelled pairs from your own documents, since no published threshold exists for this. The routing side of that decision is in [human review fallback for document extraction](/articles/human-review-fallback-for-document-extraction).

## Why does a file hash not stop duplicate records?

A file hash fails to stop duplicates because it identifies bytes, not the business document those bytes depict. Re-scanning a paper invoice, re-exporting a PDF or receiving the same document as an email attachment and as a portal download gives different bytes and a different hash for one invoice (author inference).

Two keys with different jobs fix this (author design):

| Key | Built from | Job | Fails when |
|---|---|---|---|
| Retry key | File hash, plus pipeline stage | Make a retry of the same upload safe | The same document arrives as different bytes |
| Business key | Supplier, document number, document date | Recognise the same real-world document | A field is misread, missing or reissued under a new number |

Use the retry key to make the write call repeatable, as described in [retries and idempotency for AI agents](/articles/retries-and-idempotency-for-ai-agents). Use the business key for the matching ladder above. Keep both on the staged record, because the first protects you from the pipeline and the second protects you from the world (author design).

## What states should an extracted record move through?

An extracted record should move through a small, explicit state machine: received, staged, validated, then either committed, held for review or rejected. The state names are author design; what matters is that every transition has a rule and a named actor.

1. **Staged.** The record is stored with its keys and provenance. Nothing downstream can read it as truth.
2. **Validated.** Schema, arithmetic and field-presence checks passed. The article on [table and form validation](/articles/table-and-form-validation-for-document-extraction) covers the checks themselves.
3. **Held.** A rule or the matching ladder sent it to a person.
4. **Committed.** Promoted into the system of record.
5. **Rejected.** Declined with a reason code, kept for audit.

### Who may promote a record?

Only a write service with its own narrow permission, or a named human approver, may move a record from validated to committed. The extraction call holds no write permission, the rule set out in step 4 of [the pillar's system-of-record section](/articles/ai-document-extraction-pipeline-in-production#how-should-extracted-data-reach-the-system-of-record); what this article adds is that promotion is a state transition with a named actor (author design). Record the actor and the rule on every transition so the state history reads as an audit trail.

## Should a document commit as a whole or record by record?

Commit a document as a whole when its records are only meaningful together, and record by record when each stands alone. An invoice header and its line items are the usual whole-document case, since committing lines without a header, or a header whose lines fail validation, leaves the books inconsistent (author inference).

| Document shape | Commit unit | Reason |
|---|---|---|
| Invoice or order with header and lines | Whole document, in one transaction | Parts are meaningless alone |
| Batch statement of independent rows | Per record | One bad row should not block the rest |
| Mixed packet of several documents | Per document, with a packet-level status | Documents are independent, the packet is not |

If a whole-document commit fails, the whole document stays staged and nothing is half-written. If a per-record commit fails on some rows, report exactly which ones, so a retry targets only those. Whichever unit you choose, make that choice per document type and write it down; a mixed policy discovered during an incident is hard to reason about.

## What happens to writes that keep failing?

Writes that exhaust their retries go to a dead-letter queue with the error reason, and are replayed later from the saved parse artifact. This is author design and relies on keeping that artifact, which the pillar's launch checklist already advises, within your retention rules.

- **Retry transient errors.** Use bounded attempts with backoff and the same retry key. The mechanics, including what counts as transient, are in [retries and idempotency for AI agents](/articles/retries-and-idempotency-for-ai-agents).
- **Dead-letter permanent errors.** A validation failure or a rejected foreign key will not succeed on retry, so park the record with its reason rather than looping.
- **Replay from the parse.** Stage A output (the parsed structure) is the cost you avoid repeating (author inference). Replaying from the stored parse reruns only extraction or the write, and lets you fix a bug, then reprocess without paying to parse again.
- **Survive restarts.** If the write path is a multi-step workflow, [durable execution for AI agents](/articles/durable-execution-for-ai-agents) explains how to resume it after a crash (author design).

Replay is also how you apply a schema change to old documents. The compatibility rules for that are in [extraction schema versioning for document pipelines](/articles/extraction-schema-versioning-for-document-pipelines).

## What should you log at the write boundary?

Log, for every write attempt, the document identity, both keys, the matching rung that decided, the state transition with its actor, and the outcome with any error. This list is author design; the general principles of what an audit record holds are in [audit trails for AI agents](/articles/audit-trails-for-ai-agents), and are not re-derived here.

Three habits make the log useful (author inference):

- **Log the decision, not just the result.** Recording that a record matched on rung two with a stated score lets you audit and tune thresholds later.
- **Keep a link from the committed row back to the staged row.** An auditor should be able to walk from a figure in the ledger to the page that produced it.
- **Do not log the document contents unnecessarily.** The log should hold identifiers and reasons, because copying full documents into logs widens your data exposure.

## What does ChangeGamer know first-hand about this?

As of October 2026 ChangeGamer runs no document extraction, so it has no write-side incidents, throughput figures or match rates to report. Treat this article as a design method, and test every threshold, key and commit unit on your own labelled documents before any record reaches a ledger.

## Frequently asked questions

### Why is a file hash not enough to prevent duplicate records?

A file hash only recognises byte-identical files. A re-scan, a re-export or a re-sent PDF of the same invoice has different bytes and so a different hash, which means duplicates slip through unless you also check a business key such as supplier, document number and date (author design).

### Who should be allowed to promote an extracted record into the database?

Only a write service with its own narrowly scoped permissions, or a named human approver, should promote a record, and each promotion should log the actor and the rule that allowed it. That keeps a wrong or manipulated extraction from becoming a committed change without passing validation, matching and, where needed, human approval (author design).

### What should happen when a write to the system of record fails?

Retry transient failures with a bounded number of attempts using the same idempotency key, and send records that exhaust their attempts or fail permanently to a dead-letter queue with the error reason. Replay them later from the saved parse artifact instead of re-parsing the document (author design).

### How do I match an extracted invoice to an existing record?

Look up the exact business key first, then compare near matches on normalised fields such as supplier name and amount, and send ambiguous cases to a person. Never auto-merge on a fuzzy match alone, because a wrong merge silently overwrites a correct record (author design).

### Does ChangeGamer run a document extraction pipeline?

No. As of October 2026 ChangeGamer runs no document extraction, so everything here is design guidance and inference built on its reference corpus, not measured results from a production pipeline of its own.


---

## The rest of this guide

- [AI Document Extraction in Production: Parse, Extract, Validate, Review](https://changegamer.ai/articles/ai-document-extraction-pipeline-in-production.md): How to run AI document extraction as a production pipeline: separate parsing from schema-constrained field extraction, choose OCR, a vision model or a hybrid, validate values, route doubtful ones to a human, version the schema, and work out cost per page.
- [OCR vs Vision Language Model for Document Parsing: How to Choose Stage A](https://changegamer.ai/articles/ocr-vs-vision-model-vs-hybrid-document-parsing.md): OCR vs vision language model for document parsing: a decision order keyed on residency, layout variance, volume and known schema, how a hybrid arbitrates disagreements, a per-field bake-off, failure-mode mapping, page batching and privacy.
- [Schema-Constrained Field Extraction From Documents: Settings and Failure Branches](https://changegamer.ai/articles/schema-constrained-field-extraction-from-documents.md): Schema constrained field extraction from documents: nullable fields, enums with an escape value, evidence fields, a branch table for refusal, truncation, parse failure and empty output, a one-retry policy, and field confidence as inference.
- [Table and Form Validation for Document Extraction: Row Counts, Totals and Required Fields](https://changegamer.ai/articles/table-and-form-validation-for-document-extraction.md): Validating table and form extraction from documents: rows split across pages, subtotal rows, exact money reconciliation, column sums, required-field maps per form type, cross-field checks and a failed-check-to-route table.
- [Human Review Fallback for Document Extraction: Triggers, Capacity and the Correction Loop](https://changegamer.ai/articles/human-review-fallback-for-document-extraction.md): Designing the human review fallback for document extraction: a routing-trigger table, cross-checking high-stakes fields, review capacity and backlog policy, the review item record and a correction loop that feeds regression sets.
- [Versioning a Document Extraction Schema: Migrations, Manifests and Rollback](https://changegamer.ai/articles/extraction-schema-versioning-for-document-pipelines.md): How to version a document extraction schema: which edits break stored records, absent versus null fields, lazy upcast versus backfill, reprocessing from stored parse output, a version manifest, dual-read cutover, correction migration and rollback.
- [Document Extraction Cost per Page: Tokens, Retries, Review and Budget Alarms](https://changegamer.ai/articles/cost-per-page-for-document-extraction.md): How to model document extraction cost per page beyond the parse fee: page-count mix versus per-page pricing, image tokens versus per-call overhead, retries by failure class, review share, batch versus sync, and a budget kill switch on cost per accepted document.
- [Open Source vs Managed Document Parsing: A Decision Order, Operating Burden and Exit Plan](https://changegamer.ai/articles/open-source-vs-managed-document-parsing.md): How to decide between self-hosted open-source parsers and a managed parsing service: residency and licence first, volume and operations capacity second, reversibility third, plus the cost of operating models, lock-in, a single-interface fallback design and a trial protocol on your own documents.
- [Multimodal Document Understanding for Extraction: Charts, Diagrams, Photos, Stamps and Video Frames](https://changegamer.ai/articles/multimodal-document-understanding-for-extraction-pipelines.md): How to handle non-text visuals in a document extraction pipeline: when OCR is not enough, extracting a value versus a description versus a class, region-level routing, figure-aware page packing, poor scans, video frames, image-borne injection and per-visual evaluation.
- [Monitoring Document Extraction Accuracy in Production: Audit Samples, Field Metrics and Drift](https://changegamer.ai/articles/monitoring-document-extraction-accuracy-in-production.md): How to monitor document extraction accuracy in production: a stratified audit sample with independent labels, per-field metrics with intervals, extraction-specific drift triggers, thresholds from your own variance, and regression intake.
- [How to Split Long Documents for Extraction: Batches, Overlap and Reassembly](https://changegamer.ai/articles/splitting-long-documents-for-document-extraction.md): How to split long documents for extraction: size batches from measured tokens per page, choose section or page windows, overlap pages, carry table headers, reconcile conflicts across batches and reassemble with provenance.

## Reference resources

- https://changegamer.ai/resources/document-extraction-for-agents.md
- https://changegamer.ai/resources/structured-outputs-and-json-mode.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt
Licensing: https://changegamer.ai/api/pricing.json (offer catalog) · https://changegamer.ai/api/payment.json (payment methods, HTTP 402 flow) · access guide: https://changegamer.ai/resources/access-and-pricing.md
