Writing Extracted Document Data to a System of Record: Matching, Keys, States and Replay
How to write extracted document data to a system of record: matching an extracted record to an existing one, retry keys versus business keys, a staged-to-committed state machine, per-record versus per-document atomicity, dead-letter queues, replay from the saved parse artifact and the write-boundary log.
- A file hash identifies one upload and makes retries safe, but a re-scan of the same paper document produces different bytes and a different hash, so duplicate detection also needs a business key built from fields such as supplier, document number and date (author design).
- Matching an extracted record to an existing record works best as a ladder: exact business-key lookup first, fuzzy comparison second, and a human decision for anything the first two cannot settle with confidence (author design).
- A staged-to-committed state machine for extracted records should let only code with its own write permission, or a named human reviewer, promote a record, and never the extraction call itself (author design).
- Saving the parse artifact for every document lets a failed or rejected write be replayed from the stored parse without paying for parsing again, and a dead-letter queue holds the records that exhausted their retries (author design).
- Whether a document commits atomically as a whole or record by record depends on whether its records are meaningful alone; an invoice header and its lines usually are not, so they commit together (author inference).
Write extracted document data to a system of record through code that is separate from the model: match each extracted record to an existing one, stage it, validate it, and promote it under a rule about who may. The AI document extraction pipeline pillar already covers the basics in its system-of-record section: staging, an idempotency key, provenance, and no direct writes. This piece does not repeat them. It covers the decisions that follow once the basics are in place. The extraction call itself returns a record (author inference; see Structured Outputs and JSON Mode for the output-shape mechanics), and Document Extraction and Parsing for Agents (2026-09-28) describes the parsing stage that precedes it. Everything on the write side is author design or inference, labelled as such. Nothing here is legal advice.
Match in a ladder of three rungs: an exact business-key lookup, then a fuzzy comparison, then a human decision for whatever remains. This ladder is author design, not a published standard, and it exists because each rung is cheaper and safer than the next.
| Rung | Method | Outcome | Risk if misused |
|---|---|---|---|
| 1. Exact | Look up the business key (for example supplier, document number, date) | Same record found, or none | Low; a normalisation gap makes a miss look like a new record |
| 2. Fuzzy | Compare normalised names, amounts and dates within a tolerance | Candidate matches with a score | A wrong merge overwrites a correct record |
| 3. Human | A reviewer sees the extracted record beside the candidates | A decision with a named approver | Slow, so reserve it for real ambiguity |
Normalise before every comparison: trim whitespace, fold case, strip punctuation from identifiers and parse dates into one format. Most exact-match misses on identical documents come from formatting differences, not from a different document (author inference).
A fuzzy match should never merge records on its own; it may only propose candidates for a person or a stricter rule to confirm. Treat the score threshold as something you calibrate on labelled pairs from your own documents, since no published threshold exists for this. The routing side of that decision is in human review fallback for document extraction.
A file hash fails to stop duplicates because it identifies bytes, not the business document those bytes depict. Re-scanning a paper invoice, re-exporting a PDF or receiving the same document as an email attachment and as a portal download gives different bytes and a different hash for one invoice (author inference).
Two keys with different jobs fix this (author design):
| Key | Built from | Job | Fails when |
|---|---|---|---|
| Retry key | File hash, plus pipeline stage | Make a retry of the same upload safe | The same document arrives as different bytes |
| Business key | Supplier, document number, document date | Recognise the same real-world document | A field is misread, missing or reissued under a new number |
Use the retry key to make the write call repeatable, as described in retries and idempotency for AI agents. Use the business key for the matching ladder above. Keep both on the staged record, because the first protects you from the pipeline and the second protects you from the world (author design).
An extracted record should move through a small, explicit state machine: received, staged, validated, then either committed, held for review or rejected. The state names are author design; what matters is that every transition has a rule and a named actor.
- Staged. The record is stored with its keys and provenance. Nothing downstream can read it as truth.
- Validated. Schema, arithmetic and field-presence checks passed. The article on table and form validation covers the checks themselves.
- Held. A rule or the matching ladder sent it to a person.
- Committed. Promoted into the system of record.
- Rejected. Declined with a reason code, kept for audit.
Only a write service with its own narrow permission, or a named human approver, may move a record from validated to committed. The extraction call holds no write permission, the rule set out in step 4 of the pillar's system-of-record section; what this article adds is that promotion is a state transition with a named actor (author design). Record the actor and the rule on every transition so the state history reads as an audit trail.
Commit a document as a whole when its records are only meaningful together, and record by record when each stands alone. An invoice header and its line items are the usual whole-document case, since committing lines without a header, or a header whose lines fail validation, leaves the books inconsistent (author inference).
| Document shape | Commit unit | Reason |
|---|---|---|
| Invoice or order with header and lines | Whole document, in one transaction | Parts are meaningless alone |
| Batch statement of independent rows | Per record | One bad row should not block the rest |
| Mixed packet of several documents | Per document, with a packet-level status | Documents are independent, the packet is not |
If a whole-document commit fails, the whole document stays staged and nothing is half-written. If a per-record commit fails on some rows, report exactly which ones, so a retry targets only those. Whichever unit you choose, make that choice per document type and write it down; a mixed policy discovered during an incident is hard to reason about.
Writes that exhaust their retries go to a dead-letter queue with the error reason, and are replayed later from the saved parse artifact. This is author design and relies on keeping that artifact, which the pillar's launch checklist already advises, within your retention rules.
- Retry transient errors. Use bounded attempts with backoff and the same retry key. The mechanics, including what counts as transient, are in retries and idempotency for AI agents.
- Dead-letter permanent errors. A validation failure or a rejected foreign key will not succeed on retry, so park the record with its reason rather than looping.
- Replay from the parse. Stage A output (the parsed structure) is the cost you avoid repeating (author inference). Replaying from the stored parse reruns only extraction or the write, and lets you fix a bug, then reprocess without paying to parse again.
- Survive restarts. If the write path is a multi-step workflow, durable execution for AI agents explains how to resume it after a crash (author design).
Replay is also how you apply a schema change to old documents. The compatibility rules for that are in extraction schema versioning for document pipelines.
Log, for every write attempt, the document identity, both keys, the matching rung that decided, the state transition with its actor, and the outcome with any error. This list is author design; the general principles of what an audit record holds are in audit trails for AI agents, and are not re-derived here.
Three habits make the log useful (author inference):
- Log the decision, not just the result. Recording that a record matched on rung two with a stated score lets you audit and tune thresholds later.
- Keep a link from the committed row back to the staged row. An auditor should be able to walk from a figure in the ledger to the page that produced it.
- Do not log the document contents unnecessarily. The log should hold identifiers and reasons, because copying full documents into logs widens your data exposure.
As of October 2026 ChangeGamer runs no document extraction, so it has no write-side incidents, throughput figures or match rates to report. Treat this article as a design method, and test every threshold, key and commit unit on your own labelled documents before any record reaches a ledger.
Frequently asked questions
- Why is a file hash not enough to prevent duplicate records?
- A file hash only recognises byte-identical files. A re-scan, a re-export or a re-sent PDF of the same invoice has different bytes and so a different hash, which means duplicates slip through unless you also check a business key such as supplier, document number and date (author design).
- Who should be allowed to promote an extracted record into the database?
- Only a write service with its own narrowly scoped permissions, or a named human approver, should promote a record, and each promotion should log the actor and the rule that allowed it. That keeps a wrong or manipulated extraction from becoming a committed change without passing validation, matching and, where needed, human approval (author design).
- What should happen when a write to the system of record fails?
- Retry transient failures with a bounded number of attempts using the same idempotency key, and send records that exhaust their attempts or fail permanently to a dead-letter queue with the error reason. Replay them later from the saved parse artifact instead of re-parsing the document (author design).
- How do I match an extracted invoice to an existing record?
- Look up the exact business key first, then compare near matches on normalised fields such as supplier name and amount, and send ambiguous cases to a person. Never auto-merge on a fuzzy match alone, because a wrong merge silently overwrites a correct record (author design).
- Does ChangeGamer run a document extraction pipeline?
- No. As of October 2026 ChangeGamer runs no document extraction, so everything here is design guidance and inference built on its reference corpus, not measured results from a production pipeline of its own.
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.