# Agentic RAG in Production: The Complete Operator Guide

> The operator playbook for agentic RAG in production: ingestion, chunking, hybrid retrieval, reranking, evaluation, freshness and cost.

Guide: RAG in production (pillar)
Published: 2026-08-22 · Updated: 2026-09-04 · 4454 words · ~5924 tokens (estimate)
Canonical: https://changegamer.ai/articles/rag-in-production
JSON: https://changegamer.ai/api/articles/rag-in-production.json

## In short

- A RAG demo and a production RAG system differ in six places: ingestion is a pipeline instead of a script, chunking is tuned to real queries instead of a copied default, the corpus changes under you (updates, deletes, permission changes), retrieval is hybrid and reranked instead of a bare top-k vector lookup, quality is measured continuously instead of eyeballed, and cost and latency are engineered instead of ignored. Most "RAG failed us" stories are a failure in one of those six places, not in the model.
- Version every component that changes query results — the embedding model above all. Re-embedding a corpus is a full reindex with a migration window where two indexes run side by side; if you do not plan for that on day one you will eventually freeze an outdated embedding model out of fear while your corpus drifts away from it.
- Hybrid retrieval (lexical plus vector, fused) is the sane production default, not an optimization. Vector-only retrieval reliably misses exact identifiers — error codes, SKU numbers, citation names — that lexical match catches trivially, and no amount of embedding-quality tuning fully closes that gap.
- Evaluation is what makes RAG maintainable: a fixed golden question set with expected source passages, retrieval metrics (recall@k, MRR) reported separately from generation metrics (faithfulness, answer relevance), and a regression gate in CI so an innocent-looking change to chunking, prompts or models cannot silently degrade answers.
- Freshness and permissions are correctness requirements, not features. A retriever that serves deleted, superseded or permission-revoked documents produces confident, fluent, wrong answers — worse than a retrieval miss, because nobody audits an answer that sounds right.
- Know when RAG is the wrong tool. If the knowledge fits in context, include it directly; if the answer lives in structured data, query it instead of retrieving prose about tables; if you need a behavior change rather than knowledge injection, fine-tuning is the lever. RAG earns its operational cost only when the corpus exceeds the context window, changes faster than release cycles, and answers demand per-query sourcing.

---


## What is agentic RAG, and what does production change about it?

Agentic RAG treats retrieval as a tool an agent calls on its own — deciding whether to retrieve at all, formulating its own queries, reading the results, and issuing follow-ups — instead of the fixed retrieve-then-generate pipeline a demo runs once per question. Every RAG tutorial produces the same artifact regardless of that distinction: a script that loads some documents, splits them into chunks, embeds them, retrieves the top few for one sample question, and stuffs them into a prompt. The demo works — and the gap between that demo and a system real users depend on is where most projects quietly die. The gap is almost never the model. The gap is that a demo runs once against a frozen corpus and a hand-picked question, while production runs forever against a corpus that keeps changing, queries nobody predicted, tenants that must not see each other's data, and a latency and cost budget somebody actually pays.

Naming the deltas explicitly makes the work plannable. Moving from demo to production means:

1. **Ingestion becomes a pipeline.** Documents arrive continuously, arrive dirty, get updated, get deleted, and carry permissions. A one-shot loader cannot express any of that.
2. **Chunking gets tuned against real queries**, not a default value copied from a quickstart.
3. **The corpus mutates under the index.** Updates, deletions and permission revocations must propagate, or the system confidently cites documents that no longer exist or should no longer be visible.
4. **Retrieval becomes layered**: hybrid lexical-plus-vector matching, metadata filtering at query time, and a reranking stage with its own budget.
5. **Quality is measured continuously**, with retrieval metrics separated from generation metrics, and regressions blocking deploys instead of being discovered by an angry user.
6. **Cost and latency are engineered**: caching, batching, smaller models where they suffice, and a budget per query.

This guide walks through each layer roughly in the order their failures arrive, at the level of decisions and trade-offs. Concept-level background — what an embedding is, how similarity search behaves — lives in the reference corpus ([RAG & retrieval for agents](/resources/rag-retrieval-for-agents), [embeddings & vector search](/resources/embeddings-vector-search)) and is deliberately not restated here. Where a section deserves deeper treatment than a pillar can give, it becomes its own article in this cluster; the cluster listing above and below this page tracks what exists.

## Why is ingestion a pipeline, not a script?

The first production surprise is almost always ingestion volume and messiness. Real corpora contain PDFs with broken layouts, wikis with stale pages, ticket systems with sensitive strings, repositories with generated artifacts, and web scrapes full of boilerplate. Extraction deserves more respect than it usually gets: if the extractor silently mangles tables, drops footnotes or merges columns, no downstream sophistication recovers information that was never captured. Treat [document extraction](/resources/document-extraction-for-agents) as a tested component with fixtures for your nastiest real documents, not as a library call. For web-sourced corpora the same applies double — boilerplate stripping, canonicalization and deduplication decide whether "similar pages at slightly different URLs" become three conflicting chunks or one clean fact ([web data for agents](/resources/web-data-for-agents)).

Four properties separate a production ingestion pipeline from a script:

- **Idempotency.** Re-running ingestion for the same document version must produce the same result without duplicating chunks. The standard mechanism is deterministic chunk IDs derived from source ID, document version and chunk ordinal, so re-ingestion upserts rather than appends. Every "why does my retrieval show the same paragraph three times" incident traces back here.

- **Incremental updates.** Re-processing an entire corpus on every change does not survive contact with real scale. Content hashing lets you skip unchanged documents cheaply; change feeds let you react near-real-time for sources that support push semantics. Document each connector's update contract — push versus poll, full versus delta — because undocumented update semantics are how silent staleness begins.

- **Deletes and tombstones.** Deletion is where naive pipelines fail hardest precisely because nothing triggers them: a removed upstream document simply stops sending events. You need either periodic reconciliation sweeps comparing source inventory against indexed inventory, or an explicit delete feed. In regulated contexts, deletion propagation carries compliance deadlines — treat it as a correctness requirement with an SLA, not as hygiene.

- **Durability and backpressure.** Ingestion jobs die midway through large batches, embedding APIs rate-limit, vector stores briefly reject writes. A queue with retries, dead-letter handling and resumable checkpoints turns those from incidents into log lines. If your orchestrator cannot resume a half-finished batch after a crash, you will eventually re-index from zero at the worst possible time; durable execution patterns exist exactly for this class of workflow ([handling rate limits and retries](/resources/handling-rate-limits-and-retries), [durable execution](/resources/durable-execution-for-agents)).

One more decision that is much cheaper at ingest time than retrofitted: attach rich metadata to every chunk — source system, document type, language, publication date, access-control groups, tenant ID. Query-time filtering and permission enforcement are only ever as good as the metadata you thought to preserve before indexing.

## How do you choose a chunking strategy?

Chunking decides retrieval quality, and it is also where the least defensible defaults live. A fixed ~512-token chunk with ~50-token overlap is a reasonable starting point and a terrible ending point, because the right chunk depends entirely on what a unit of evidence must look like for your users' questions.

The decision hierarchy that works in practice:

- **Structural boundaries first.** If documents have headings, sections, list items or table rows, split along them. Structure is a free coherence signal: a section-sized chunk reads as a complete thought far more often than a fixed-window slice does.
- **Fixed windows only for genuinely unstructured prose**, sized so a typical chunk holds one complete idea plus enough surrounding context to interpret it. Overlap exists to keep boundary-spanning ideas retrievable; extra overlap costs storage and dedup effort, not just tokens.
- **Escape the precision/context dilemma with expansion.** Small chunks localize evidence sharply but starve the generator of context; large chunks carry context but blur retrieval signals. Retrieve small units, then expand to enclosing structure (the surrounding section, or a parent document) at prompt-assembly time. This gets sharp matching and generous context simultaneously.
- **Handle tables and code specially.** A table sliced mid-row is noise; keep tables whole together with their captions, and keep code aligned to function or class boundaries where possible.

Whatever you choose, wire the choice to measurement: hold a golden set of questions with known-relevant passages and compare recall@k across candidate chunkings. Chunk-size debates without a retrieval metric attached are aesthetics. And remember that changing chunking implies re-embedding and re-indexing everything — schedule it like a migration, not an edit. The strategy-level detail (recursive splitters, proposition extraction, layout-aware splitting and their failure modes) has its own reference ([chunking strategies for RAG](/resources/chunking-strategies-for-rag)).

## How should you choose and version embedding models?

Embedding selection looks like a leaderboard problem and is actually an operations problem. Benchmark deltas between leading models are small next to the operational properties that dominate total cost of ownership: dimensionality (storage and index memory scale with it), inference cost and throughput at your corpus size, language coverage relative to your actual content, maximum input length relative to your chunks, and whether the model will still be available, unchanged, a year from now.

Two rules prevent most embedding mistakes:

**Never mix embedding spaces.** Vectors are comparable only within the model and revision that produced them. A single index must contain vectors from exactly one embedding model; switching models creates a new index, not an update. Record the model identity alongside every index so the invariant is checkable.

**Plan the reindex migration on day one.** Changing models means re-embedding every document and rebuilding every index — real time and money at scale. Production systems run old and new indexes side by side during migration: dual-write during backfill, shadow-compare retrieval quality on live query traffic or the golden set, then cut over and retire the old index. Teams that skip this planning freeze on aging models because reindexing feels dangerous — which it is, exactly proportional to how unplanned it is.

Compression techniques (dimensionality reduction, quantization) trade a little recall for substantial storage and speed wins, and belong in the same measured harness as every other retrieval decision — adopt them when index cost shows up in the budget, not speculatively. The concept layer behind these choices is covered in the [embeddings & vector search reference](/resources/embeddings-vector-search).

## Why is hybrid retrieval the production default?

Vector-only retrieval has a well-documented blind spot that keeps generating incidents: exact tokens. Error codes, part numbers, API method names, statute citations, acronyms, names — queries hinging on a precise string are where dense retrieval is weakest, because embedding smooths away the very distinction the user cares about. Lexical retrieval (BM25 and relatives) handles exact strings trivially and fails on paraphrase, which dense retrieval handles trivially. The failure modes are complementary, which is why running both and fusing results is the production default rather than an enhancement ([hybrid search for RAG](/resources/hybrid-search-for-rag)). Reciprocal rank fusion is the common workhorse; weighted score fusion needs calibration but gives finer control. What matters more than technique is that neither channel is treated as optional until measurement proves your corpus never contains exact-identifier queries — and in practice it always does.

## How do metadata filters and tenant permissions work in retrieval?

Two query-time concerns belong together because they fail the same way when bolted on late.

**Metadata filtering.** Date ranges, document types, languages and source systems narrow retrieval to the subset that can legitimately answer. The operational rule is to pre-filter (restrict the candidate set) rather than post-filter (retrieve globally, then drop), because post-filtering produces the classic failure where all top-k hits get filtered out and the model answers from nothing. Vector indexes differ in how they handle filtered search at scale, which makes filtering behavior a first-class selection criterion for the store itself ([choosing a vector database](/resources/choosing-a-vector-database)) — a store that degrades sharply under selective filters will betray you exactly on your most important queries.

**Tenant isolation and permissions.** In multi-tenant systems, access control must be enforced inside retrieval, not around it. The safe pattern is to encode authorization as mandatory query-time filters derived from the authenticated principal — never as prompt instructions ("only answer from documents the user may see" is not access control; the retriever already fetched what it fetched). Permission changes must propagate into index metadata on the same timescale as the underlying policy, or revoked users keep seeing cached visibility. And treat cross-tenant leakage as a security incident class of its own: test for it explicitly with negative cases in the eval set, alongside the usual [security checklist](/resources/agentic-security-checklist) review. Privacy obligations (data residency, retention, deletion requests) attach to chunks just as they do to records anywhere else in the system ([data privacy for agents](/resources/data-privacy-for-agents)).

## When does reranking pay for itself?

First-stage retrieval optimizes recall cheaply; a reranker spends more compute per candidate to fix ordering. Cross-encoder-style rerankers score each query-document pair jointly and consistently beat raw similarity scores on relevance ordering — at a cost that scales with candidates scored. That cost is manageable precisely because reranking operates on tens of items, not thousands.

The production questions are therefore budget questions, not yes/no questions:

- **How many candidates reach the reranker?** Too few and you cap achievable quality (the true best passage may not be in the candidate set); too many and latency and cost balloon for marginal gains. Plot end-metric against candidate count once and pick the knee.
- **Does every query deserve reranking?** Simple lookups answered fine by first-stage ranking waste rerank compute; ambiguous or high-stakes queries benefit most. A cheap router or score-threshold gate keeps the budget where it helps ([agent cost & latency optimization](/resources/agent-cost-latency-optimization)).
- **What does the generator actually need?** Reranking optimizes ordering, but context assembly cares about coverage and non-redundancy across the final selection. Deduplicating near-identical passages after reranking often buys more answer quality than another reranking stage would.

As with everything else in this stack: measure the reranker's contribution separately (same golden set, reranker on versus off) so its budget has to justify itself continuously ([reranking for RAG](/resources/reranking-for-rag)).

## Prompt assembly: the last mile decides the answer

Retrieval quality sets an upper bound; prompt assembly determines how much of it survives. Four assembly decisions matter most:

1. **Order and position.** Models attend unevenly across long contexts; placing the strongest evidence early and late beats burying it mid-list. When you have relevance scores, use them for placement instead of shuffling.
2. **Attribution.** Number or label each retrieved chunk so the answer can cite which chunk supports which claim. Attribution makes hallucination visible and gives evaluation something concrete to check — unattributed synthesis is where silent errors hide.
3. **Context budget discipline.** Cramming the maximum context "because it fits" degrades attention and costs money. Set a token budget per query tier, fill it deliberately, and prefer fewer well-chosen chunks over many mediocre ones. Structured output requirements (answer plus cited sources) work better when the format contract is explicit ([structured outputs & JSON mode](/resources/structured-outputs-and-json-mode)).
4. **Caching.** System prompts and instruction scaffolds repeat across every query; cache them at the provider level where supported ([prompt caching](/resources/prompt-caching-for-agents)), and cache full responses for repeated or near-duplicate questions where freshness tolerances allow ([response caching](/resources/agent-response-caching)). Both are pure margin in a system whose unit economics are otherwise retrieval-plus-generation per query.

## How do you evaluate a RAG system?

A RAG system without evaluation cannot be safely changed, because every knob — chunk size, embedding model, fusion weights, reranker depth, prompts — moves answer quality in ways nobody can eyeball. Evaluation converts those knobs from acts of faith into measured engineering. Three layers, kept strictly separate:

- **Golden set.** A curated, version-controlled set of representative questions annotated with the passages that must be retrieved (and ideally reference answers). Cover your real query distribution: exact-identifier lookups, paraphrases, multi-hop questions, empty-result cases, and — critically — permission-negative cases that must return nothing for some principals. Thirty good cases catch more regressions than three thousand sloppy ones.

- **Retrieval metrics.** Recall@k (did the needed passage make the candidate set), MRR or nDCG (how high it ranked). These are deterministic, fast, and independent of any language model, which makes them ideal CI gates. Most "the answers got worse" mysteries resolve here: generation did not degrade, retrieval stopped surfacing the evidence.

- **Generation metrics.** Faithfulness (is the answer entailed by retrieved context) and answer relevance, checked either by rubric-based human review or by LLM-as-judge patterns with calibrated prompts. Judge-based metrics are noisier and cheaper than humans; humans are the ground truth for the cases judges disagree on. Use judges for regression screening, humans for acceptance.

Wire the offline suite into CI so every change to chunking, models, prompts or retrieval parameters runs the gauntlet automatically, and treat threshold breaches as build failures ([evaluating AI agents](/resources/evaluating-ai-agents), [testing AI agents](/resources/testing-ai-agents)). Complement offline gates with online signals — user feedback, citation clicks, escalation-to-human rates — but never let online dashboards replace the offline set: by the time a dashboard shows a regression, users have been the test suite for weeks.

Close the loop with logged queries. Real user queries are the highest-value evaluation data a RAG system produces: sample them regularly, cluster them into intents, find the ones that failed (no click, negative feedback, escalation), and promote the instructive failures into the golden set. This flywheel — production traffic feeding curated eval cases feeding CI gates — is what keeps the system's quality curve rising after launch instead of decaying as the corpus and query mix drift. It also surfaces coverage gaps early: an intent cluster with consistently poor retrieval is next quarter's ingestion or chunking work, discovered by measurement rather than by complaint.

## How do you keep a RAG index fresh?

Stale answers are the most insidious RAG failure class because the system looks completely healthy while being wrong: fluent prose, correct citations — of a policy that was replaced, a price that changed, a person who left. Preventing it is an ingestion-side property, decided long before any model runs:

- **Per-source freshness contracts.** Every connector declares its staleness bound (near-real-time, hourly, daily) and the system measures actual lag against it. Unmeasured freshness claims decay silently.
- **Versioned documents.** Superseded versions are tombstoned or explicitly marked, so retrieval cannot surface two generations of the same policy and let the model average between them.
- **Deletion propagation** with reconciliation sweeps as backstop, as covered above.
- **Effective-date awareness** where content is time-sensitive: index the validity window with the chunk so query-time filtering can exclude not-yet-effective or expired material mechanically.

For agent-facing RAG specifically, freshness interacts with memory design: conversation-level facts, task state and corpus knowledge age at different rates and belong in different stores with different invalidation rules ([agent memory & context](/resources/agent-memory-context)). Observability should treat corpus lag as a first-class metric next to latency and error rate ([agent observability](/resources/agent-observability)) — "time since last successful sync per source" is the number that predicts stale-answer complaints before they arrive.

## How do you engineer RAG cost and latency?

RAG's unit economics are simple to state and easy to lose: every query pays retrieval infrastructure plus embedding inference plus generation tokens. Engineering the budget means attacking each term:

- **Cache what repeats.** Query-result caching for popular questions, semantic caching for near-duplicates, prompt-prefix caching for shared scaffolding. Hit rates decide whether these are worth their invalidation complexity — measure before building elaborate caches.
- **Right-size the models.** Route easy queries to smaller generators; reserve large models for hard or high-stakes tiers. The router only works if evaluation confirms small-model answers hold up on the easy tier — another job for the golden set.
- **Bound fan-out.** Hybrid retrieval doubles first-stage work; deep reranking multiplies scoring cost; multi-step agentic retrieval can multiply it again per step. Keep explicit budgets per stage and alarms on p95 latency, not just averages ([cost & latency optimization](/resources/agent-cost-latency-optimization)).
- **Watch the hidden lines**: vector-store storage and memory (dimensionality drives both), reindex migrations, and eval-suite compute. Capacity planning for indexes belongs in the same review cycle as model choice.

## Retrieval as a tool: agentic RAG

The pipeline described above is single-shot: query in, chunks out, answer back. The second major consumption pattern treats retrieval as a **tool** that an agent calls autonomously — deciding whether to retrieve at all, formulating queries, reading results, issuing follow-ups, and stopping when evidence suffices. Agent frameworks made this pattern mainstream, and it changes several production calculations.

- **Query formulation becomes part of the system.** Human users type what they think; agents generate queries from their own reasoning, which can be better or worse. Expose retrieval with a deliberately narrow, well-described contract — parameters for filters, source selection and k — because the tool description is now part of your retrieval interface and the agent's only guide to using it well ([reliable tool calling](/resources/reliable-tool-calling)). Log agent-issued queries separately from human ones; their distributions differ and each needs its own golden-set coverage.

- **Iterative retrieval needs budgets.** An agent that can re-query until satisfied will sometimes converge brilliantly and sometimes loop. Bound the loop: max retrieval rounds per task, max tokens across all retrievals, and a deterministic stop condition (evidence found for all sub-questions, budget exhausted, or explicit failure). Unbounded agentic retrieval is how a cost incident hides inside a feature.

- **Decomposition beats monolithic queries.** Multi-hop questions ("which of our enterprise customers lack SSO configured in region X?") fail as single dense queries but succeed when decomposed into sequential retrievals whose intermediate results inform the next hop. That decomposition can live in the agent's planning or in explicit orchestration; either way, evaluate decomposition quality separately from retrieval quality, because a correct pipeline over a bad plan still answers wrongly ([multi-agent orchestration patterns](/resources/multi-agent-orchestration-patterns)).

- **The corpus becomes attack surface twice over.** In single-shot RAG, injected instructions in retrieved text influence one answer. When an agent treats retrieved text as observations and acts on them — calling tools, sending messages, spending money — the same injection propagates into actions. Content-trust defenses stop being optional hardening and become a launch requirement: strip instruction-like content where feasible, attribute every fact to its chunk, and constrain what downstream tools the answering process may invoke ([prompt injection design patterns](/resources/prompt-injection-design-patterns)).

- **Memory and retrieval blur together.** Conversational state, task progress and learned preferences sit alongside corpus knowledge in the agent's effective context, each aging differently. Production systems keep them separate — volatile state in session storage, durable knowledge in the indexed corpus — rather than letting conversation history silently become a second, unversioned, un-evaluated retrieval layer ([agent memory & context](/resources/agent-memory-context)).

The pillar-level guidance is unchanged by agency: hybrid retrieval, measured chunking, versioned embeddings, permission filtering. What changes is that contracts get consumed by machines that read descriptions literally, iterate without mercy, and act on what they read — so interfaces tighten, budgets harden, and trust boundaries move into the retrieval path itself.

## Failure modes: a field guide

Production RAG failures cluster into recognizable classes, and each has a standard first diagnostic:

| Failure | Looks like | First diagnostic |
|---|---|---|
| Retrieval miss | Confident "no information found", or generic answer | Run the question against the golden set; check recall@k and whether the target chunk exists at all |
| Stale answer | Fluent citation of outdated content | Check source sync lag and version tombstones |
| Permission leak | Answer cites content a tenant must not see | Reproduce with permission-negative eval case; inspect filter application order |
| Context overload | Right passage present, wrong or hedged answer | Inspect assembled prompt; reduce chunk count, improve placement |
| Injection via content | Answers follow instructions embedded in retrieved text | Treat ingested content as untrusted input; apply [injection defenses](/resources/prompt-injection-design-patterns) |
| Silent degradation | Metrics drift down over weeks | Check embedding/model drift, corpus growth skewing the index, connector breakage |

Two systemic habits keep this table short. First, log retrieval traces (query, matched chunks, scores, filters applied) with the same seriousness as application logs — nearly every incident above is debuggable in minutes with traces and opaque without them. Second, run permission-negative and injection probes continuously in the eval suite, so the security-class failures surface as red builds rather than as incidents ([agentic security checklist](/resources/agentic-security-checklist)).

## When is RAG the wrong answer?

RAG has real fixed costs — pipelines, indexes, evaluations, on-call. Choosing it for problems that do not need it is how teams conclude "RAG is overcomplicated." The honest decision tree:

- **Fits in context and rarely changes?** Include it directly. No retrieval beats perfect retrieval.
- **Lives in structured data?** Query it. Text-to-SQL or API calls return exact, fresh, aggregable answers; retrieving prose about tables loses precision at every layer ([text-to-SQL agents](/resources/text-to-sql-agents)).
- **Needs behavior or style change?** Fine-tune. Fine-tuning adjusts how the model behaves; RAG supplies what it knows. They solve different problems and combine cleanly when both needs exist ([fine-tuning vs RAG](/resources/fine-tuning-vs-rag)).
- **Questions are about relationships between entities?** Consider graph approaches, which represent relations natively instead of hoping co-located prose carries them ([GraphRAG for agents](/resources/graphrag-for-agents)).
- **Corpus exceeds context, changes fast, demands sourcing?** This is RAG's home turf — proceed with everything above.

## A production readiness checklist

Before calling a RAG system production-ready, it should survive this list:

- Ingestion is idempotent, incremental, handles deletes, resumes after crashes
- Chunking strategy chosen against retrieval metrics on a representative golden set, not defaults
- One embedding model per index; model identity recorded; reindex migration path rehearsed
- Hybrid retrieval enabled; fusion method chosen by measurement; exact-identifier queries verified
- Pre-filtering used for metadata and permissions; permission-negative tests in the eval suite
- Reranking depth justified by measurement; budget bounded; deduplication applied post-rerank
- Prompt assembly uses attribution, deliberate placement and explicit token budgets
- Golden-set evals run in CI; retrieval and generation metrics reported separately; thresholds block merges
- Freshness contracts per source, measured; deletion propagation tested; corpus lag monitored
- Cost per query known; caching evaluated against hit rates; p95 latency alarmed
- Injection and content-trust defenses reviewed against the current threat notes

None of these are exotic. The consistent theme is that production RAG is a systems-engineering discipline wrapped around a statistical core: pipelines that respect data reality, retrieval tuned by measurement, correctness properties (freshness, permissions, provenance) enforced mechanically, and evaluation making every future change boring. Teams that internalize this stop experiencing RAG as fragile — and start shipping answers their users can audit, cite and trust.

## The full RAG in production cluster

Thirteen sub-articles make up the rag-in-production cluster, each going deeper on one piece of this playbook than a single guide reasonably can, and the cluster is complete as of August 2026:

- [How to build a RAG ingestion pipeline that survives production](/articles/rag-ingestion-pipeline) turns the idempotency, incremental-update and delete-propagation properties above into the actual queueing, checkpointing and reconciliation mechanics.
- [How to chunk documents for RAG](/articles/chunking-documents-for-rag) expands the structural-boundaries-first decision hierarchy above into the recursive-splitter, proposition-extraction and layout-aware techniques and their failure modes.
- [How to choose an embedding model for RAG](/articles/choosing-embedding-models) is the deeper operations case behind the "never mix embedding spaces" rule above, including the reindex-migration mechanics this pillar only sketches.
- [How to combine keyword and vector search in RAG](/articles/hybrid-retrieval-fusion) turns the hybrid-retrieval-as-default argument above into reciprocal rank fusion, weighted fusion and their calibration tradeoffs.
- [When and how to rerank retrieved documents](/articles/reranking-retrieved-results) expands the reranking budget questions above into candidate-count tuning and routing patterns.
- [How to evaluate a RAG system](/articles/evaluating-rag-systems) turns the golden-set and CI-gate outline above into concrete metric definitions, judge design and threshold-setting.
- [How to keep a RAG index fresh](/articles/rag-index-freshness) is the deeper discipline behind the freshness-contracts section above, for corpora that never stop changing.
- [Permission-aware retrieval: multi-tenant RAG without leaks](/articles/multi-tenant-rag-permissions) expands the tenant-isolation guidance above into the query-time filter architecture and negative-test design that guidance only summarizes.
- [Retrieval as a tool: agentic RAG patterns that survive production](/articles/agentic-retrieval-patterns) turns the agentic-RAG section above into concrete budget, decomposition and trust-boundary mechanics.
- [How to cut RAG cost and latency without cutting quality](/articles/rag-cost-and-latency) is the fuller structural breakdown behind the cost-engineering section above.
- [Common RAG failure modes and how to fix them](/articles/rag-failure-modes-runbook) expands the failure-mode field guide above into a complete runbook per failure class.
- [GraphRAG vs vector RAG: when to use a knowledge graph instead](/articles/graphrag-vs-vector-rag) turns the graph-approaches bullet in the decision tree above into a full comparison, cost breakdown and hybrid-retrieval framework.
- [When is RAG the wrong answer?](/articles/when-rag-is-the-wrong-answer) turns the five-branch decision tree above into a worked guide, with real depth on text-to-SQL and fine-tuning as RAG's two most common replacements.


## Frequently asked questions

### What is agentic RAG?

Agentic RAG is retrieval treated as a tool an agent calls on its own — deciding whether to retrieve at all, formulating its own queries, reading the results, issuing follow-up retrievals, and stopping once it judges the evidence sufficient — rather than the fixed retrieve-then-generate pipeline a single-shot RAG system runs once per question. It changes several production calculations: query formulation becomes part of the system instead of coming from a human, iterative retrieval needs an explicit round and token budget to avoid an unbounded loop, and injected instructions in retrieved text become an action-level risk rather than just a bad answer, because the agent can act on what it reads.

### What is the biggest difference between a RAG prototype and production RAG?

A prototype optimizes for one impressive answer on a frozen corpus; production optimizes for the distribution of answers over time while the corpus, the query mix and the models change. That shifts the engineering to idempotent ingestion pipelines, chunking tuned against observed queries, embedding-model versioning with planned reindex migrations, hybrid retrieval with reranking budgets, continuous evaluation wired into CI, freshness and deletion guarantees, per-tenant permission filtering, and explicit cost and latency targets.

### How large should chunks be for RAG?

No universal number exists: the right chunk is the smallest unit of text that stands alone as evidence for the questions your users actually ask. Teams typically start somewhere in the low hundreds of tokens with modest overlap and then tune against recall@k on their own golden set. The chunking strategy (structural versus fixed-window versus parent-document) matters more than the exact token count, and any chunking change should be validated by measurement, not by a blog-post default.

### How often should I refresh or re-index a RAG corpus?

Tie ingestion to each source's update pattern: near-real-time change capture for transactional content, scheduled crawls for slowly changing web or file sources, and immediate deletion propagation wherever removals carry legal or security weight. Separately, re-embed everything only when you switch embedding models or observe measurable retrieval decay on your eval set — and run old and new indexes in parallel during the migration window rather than cutting over blindly.

### How do I evaluate a RAG system properly?

Maintain a golden set of representative questions annotated with the passages that must be retrieved and reference answers. Score retrieval separately (recall@k, MRR or nDCG) from generation (faithfulness to the retrieved context, answer relevance). Run it in CI on every change to chunking, retrieval parameters, prompts or models, and treat a drop beyond threshold as a build failure. Online signals such as user feedback, citation clicks and escalation rates complement but never replace the offline set.

### When should I not use RAG?

Skip RAG when the knowledge fits comfortably in the context window and changes rarely (just include it), when answers live in structured data (query the database instead of retrieving prose about tables), when the goal is behavior or style rather than facts (fine-tuning is the lever), or when entity relationships are the core of the questions (a graph approach may serve better than passage retrieval). RAG adds real infrastructure cost; it pays off when the corpus exceeds the context window, changes frequently, and requires per-query provenance.


---

## Everything in this guide

- [How to Build a RAG Ingestion Pipeline That Survives Production](https://changegamer.ai/articles/rag-ingestion-pipeline.md): The six properties that separate a production RAG ingestion pipeline from a demo script: tested extraction, idempotent writes, incremental updates, deletion propagation, durable execution, and metadata captured at ingest time.
- [How to Chunk Documents for RAG (Strategy Beats Size)](https://changegamer.ai/articles/chunking-documents-for-rag.md): Chunking decisions that actually move retrieval quality: structural boundaries before fixed windows, parent-document expansion, special handling for tables and code, overlap trade-offs, and tuning against recall@k instead of blog defaults.
- [How to Choose an Embedding Model for RAG (and Version It Like a Schema)](https://changegamer.ai/articles/choosing-embedding-models.md): Embedding selection as an operations problem: the criteria that dominate total cost of ownership, the never-mix-spaces invariant, reindex migrations with dual indexes, and where quantization fits once cost shows up.
- [How to Combine Keyword and Vector Search in RAG](https://changegamer.ai/articles/hybrid-retrieval-fusion.md): Why hybrid retrieval is the production default rather than an upgrade: complementary failure modes of lexical and dense search, reciprocal rank fusion versus weighted scoring, parameter choices, and how filtering interacts with fusion.
- [When and How to Rerank Retrieved Documents in RAG](https://changegamer.ai/articles/reranking-retrieved-results.md): Reranking as a budget decision: why first-stage ranking misorders good evidence, when cross-encoder reranking pays for itself, how to pick candidate depth at the knee, gating by query difficulty, and deduplicating after fusion.
- [How to Evaluate a RAG System (Retrieval Metrics, Generation Metrics, CI Gates)](https://changegamer.ai/articles/evaluating-rag-systems.md): The evaluation harness that keeps RAG changeable: golden-set construction, retrieval metrics separated from generation metrics, LLM-as-judge screening with human acceptance, CI regression gates, and the logged-query flywheel.
- [How to Keep a RAG Index Fresh (Staleness Bounds, Not Vibes)](https://changegamer.ai/articles/rag-index-freshness.md): Freshness as an engineered property: per-source staleness contracts, document versioning and tombstones, effective-date filtering, deletion propagation with reconciliation backstops, and the sync-lag metrics that predict stale answers before users report them.
- [Permission-Aware Retrieval: Multi-Tenant RAG Without Leaks](https://changegamer.ai/articles/multi-tenant-rag-permissions.md): How to enforce access control inside retrieval for multi-tenant RAG: authorization as mandatory pre-filters derived from the authenticated principal, tenant isolation mechanics, permission-negative testing, and the post-filter trap that produces empty answers.
- [Retrieval as a Tool: Agentic RAG Patterns That Survive Production](https://changegamer.ai/articles/agentic-retrieval-patterns.md): When AI agents consume retrieval as a tool rather than a pipeline stage: narrow tool contracts, hard budgets on iterative retrieval, query decomposition, provenance-carrying results, and keeping trust boundaries inside the retrieval path.
- [How to Cut RAG Cost and Latency Without Cutting Quality](https://changegamer.ai/articles/rag-cost-and-latency.md): RAG cost and latency as engineered budgets: where the money actually goes, caching layers and their hit-rate economics, routing queries to right-sized models, bounding retrieval fan-out, the hidden lines (reindex migrations, eval compute), and p95 discipline.
- [Common RAG Failure Modes and How to Fix Them](https://changegamer.ai/articles/rag-failure-modes-runbook.md): An operator runbook for the four RAG failure classes with no dedicated deep-dive elsewhere: retrieval miss, context overload, injection via content at ingestion time, and silent quality degradation — symptom, first diagnostic, and fix for each.
- [GraphRAG vs Vector RAG: When to Use a Knowledge Graph Instead](https://changegamer.ai/articles/graphrag-vs-vector-rag.md): A decision framework for choosing graph-structured retrieval over standard vector RAG: which query types GraphRAG actually wins, what building a knowledge graph costs, named implementations, and hybrid vector-plus-graph patterns.
- [When Is RAG the Wrong Answer? A Decision Guide](https://changegamer.ai/articles/when-rag-is-the-wrong-answer.md): A worked decision guide for the four real alternatives to retrieval-augmented generation: including knowledge directly, querying structured data with text-to-SQL, fine-tuning for behavior change, and graph-based retrieval for entity relationships.

## Reference resources

- https://changegamer.ai/resources/rag-retrieval-for-agents.md
- https://changegamer.ai/resources/chunking-strategies-for-rag.md
- https://changegamer.ai/resources/embeddings-vector-search.md
- https://changegamer.ai/resources/hybrid-search-for-rag.md
- https://changegamer.ai/resources/reranking-for-rag.md
- https://changegamer.ai/resources/choosing-a-vector-database.md
- https://changegamer.ai/resources/graphrag-for-agents.md
- https://changegamer.ai/resources/fine-tuning-vs-rag.md
- https://changegamer.ai/resources/document-extraction-for-agents.md
- https://changegamer.ai/resources/web-data-for-agents.md
- https://changegamer.ai/resources/agent-memory-context.md
- https://changegamer.ai/resources/evaluating-ai-agents.md
- https://changegamer.ai/resources/testing-ai-agents.md
- https://changegamer.ai/resources/agent-cost-latency-optimization.md
- https://changegamer.ai/resources/prompt-caching-for-agents.md
- https://changegamer.ai/resources/agent-response-caching.md
- https://changegamer.ai/resources/handling-rate-limits-and-retries.md
- https://changegamer.ai/resources/durable-execution-for-agents.md
- https://changegamer.ai/resources/data-privacy-for-agents.md
- https://changegamer.ai/resources/agentic-security-checklist.md
- https://changegamer.ai/resources/prompt-injection-design-patterns.md
- https://changegamer.ai/resources/text-to-sql-agents.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt
Licensing: https://changegamer.ai/api/pricing.json (offer catalog) · https://changegamer.ai/api/payment.json (payment methods, HTTP 402 flow) · access guide: https://changegamer.ai/resources/access-and-pricing.md