How to Build a RAG Ingestion Pipeline That Survives Production
The six properties that separate a production RAG ingestion pipeline from a demo script: tested extraction, idempotent writes, incremental updates, deletion propagation, durable execution, and metadata captured at ingest time.
- A production ingestion pipeline is defined by six properties: extraction you test against real documents, idempotent writes with deterministic IDs, incremental processing per source, explicit delete propagation with reconciliation backstops, durable execution that resumes after crashes, and rich metadata attached at ingest time. Everything downstream — chunking quality, filter correctness, permission enforcement — inherits its ceiling from this layer.
- Deterministic chunk IDs (source ID + document version + chunk ordinal) are what make re-ingestion an upsert instead of a duplicate factory. Every "the same paragraph appears three times" incident traces to generated or random identifiers.
- Deletes are the failure mode nothing triggers: an upstream removal simply stops sending events. Without periodic inventory reconciliation or a delete feed, your index silently answers from documents that no longer exist.
- Attach access-control groups, tenant ID, source system, language and publication date to every chunk at ingest time. Query-time filtering is only as good as the metadata you preserved before indexing — retrofitting it means a full reprocess.
The pillar guide names ingestion as the first demo-to-production delta: scripts load files once, while systems must process documents that arrive continuously, change without notice, carry permissions, and occasionally vanish. This article is the build sheet for that layer. The pattern that holds across teams is blunt: retrieval failures get debugged first, but the root cause usually lives upstream, in ingestion.
What does extraction need to prove before you trust it?
Extraction converts PDFs, wikis, tickets, code repositories and web pages into text worth indexing. It deserves fixtures, not faith. Collect the nastiest real documents you have — multi-column layouts, scanned tables, footnotes, nested lists — and write tests asserting what survives extraction. If tables collapse into gibberish or footnote text merges into body prose, no amount of chunking cleverness recovers information that was already lost. Teams that skip this stage discover it later as mysterious retrieval misses concentrated on one document class.
Web-sourced corpora add canonicalization work: boilerplate stripping, near-duplicate collapsing, and URL normalization decide whether three variants of one page become three conflicting chunks or one clean fact. Treat extraction and normalization as components with owners and tests, not library calls at the top of a script.
Which properties make the pipeline production-grade?
Six properties separate pipelines that survive from scripts that demo:
- Idempotent writes. Deterministic chunk IDs derived from source ID, document version and chunk ordinal turn re-ingestion into an upsert. Re-run safety is non-negotiable once schedules and retries exist, because both will re-run things.
- Incremental processing. Content hashing skips unchanged documents cheaply; change feeds replace polling where sources support push semantics. Each connector gets a documented update contract: push or poll, full or delta.
- Delete propagation. Explicit delete handling for sources that emit removals, plus scheduled reconciliation sweeps comparing upstream inventory against indexed inventory. Tombstones mark removed content so retrieval can never resurface it.
- Durable execution. Queues with retries, dead-letter handling and resumable checkpoints convert crashes and rate limits into log lines instead of incidents.
- Backpressure awareness. Embedding APIs throttle; vector stores briefly reject writes. Pipelines that cannot slow down will corrupt their own tail batches under load.
- Metadata capture at ingest. Source system, document type, language, dates, tenant ID and ACL groups attach to every chunk before indexing, because none of it can be reconstructed afterward without a full reprocess.
None of these are exotic individually. The discipline is refusing to call the layer done until all six hold, because each missing property produces a specific, recurring production failure.
How should updates actually flow?
Model each source's update semantics explicitly. Transactional stores offer change capture with near-real-time lag; file stores and wikis need scheduled crawls with hash-based diffing; external web sources need crawl budgets and politeness windows. Per-source staleness bounds — "this source is never more than N hours stale" — turn freshness from an aspiration into something monitorable, because the pipeline can measure actual sync lag against the declared bound and alarm on drift.
Version documents explicitly: a new version of a policy supersedes, rather than joins, the old one. Superseded versions get tombstoned or flagged so retrieval cannot surface two generations side by side and let the generator blend them into a confident contradiction.
What does durability require in practice?
Batch jobs die midway, embedding providers return sustained 429s, and databases restart. A durable pipeline assumes all three will happen this month. Concretely:
- Work flows through a queue with visibility timeouts, not an in-memory loop
- Every batch checkpoint records what completed, so resumes continue rather than restart
- Dead-letter queues isolate poison documents for inspection instead of blocking the lane
- Rate-limit responses trigger backing off and retrying with jitter, not error storms
Workflow engines and durable-execution runtimes encode exactly these guarantees; adopting one early costs less than rebuilding after the first 3 a.m. re-index-from-zero.
Where do validation gates belong between stages?
Treat each pipeline stage as having an output contract, and check it mechanically before items flow onward:
- Extraction gates on minimum text yield per document type and on structural markers surviving (table row counts, heading counts versus the source)
- Chunking gates on size-distribution sanity: a batch whose chunks are all half the target length signals a splitter regression, not bad luck
- Embedding gates on batch success rate and vector norm drift, which catches silent model or version swaps
- Index writes gate on acknowledged acks from the store, never fire-and-forget
Quarantine documents that fail a gate into a dead-letter path with the failure reason attached, so fixes are targeted reprocessings rather than full-corpus reruns. The theme is boring but decisive: every stage can emit garbage, so every stage proves its output before the next stage spends money on it.
How do you know ingestion is healthy?
Monitor ingestion with the same seriousness as the serving path: documents processed and skipped per run, embedding cost per batch, queue depth and age, dead-letter count, per-source sync lag against its declared bound, and reconciliation mismatch counts. Corpus lag — time since last successful sync per source — is the single number that best predicts "stale answer" complaints before users file them. An ingestion layer with these signals is boring, which is precisely the goal. The final habit is post-incident review routed back into fixtures: every extraction or ingestion incident becomes a test case in the suite, so the pipeline accumulates immunity to the documents that actually hurt it.
Frequently asked questions
- How do I make RAG ingestion idempotent?
- Derive every chunk ID deterministically from the source document identity — typically source ID plus document version plus chunk ordinal — so re-running ingestion for the same version produces identical keys. Writes then become upserts keyed on that ID, and re-ingesting a document replaces rather than duplicates its chunks. Random or auto-generated IDs break this property and are the standard cause of duplicated retrieval results.
- Should I re-embed the whole corpus when one document changes?
- No. Hash each raw document at ingest and skip any whose hash matches the stored value, so only changed documents flow through chunking and embedding. For sources that support it, consume change events instead of polling full inventories. Full re-processing is reserved for model or chunking-strategy migrations, which should be run as planned dual-index migrations rather than ad-hoc rebuilds.
- How do I handle deleted documents in a RAG index?
- Use two mechanisms together: an explicit delete path for sources that emit removals, and periodic reconciliation sweeps that compare the upstream document inventory against the indexed inventory and tombstone anything missing upstream. Reconciliation is the backstop that catches silent deletions, which are the most common staleness source because nothing triggers them.
- What metadata should I store with each chunk?
- At minimum: source system, document identifier and version, document type, language, publication or effective date, tenant or workspace ID, and access-control groups. These power query-time filters and permission enforcement later. Metadata that was not captured at ingest cannot be recovered without reprocessing the corpus, so err on the side of storing more.
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.