← All guides · RAG in production

Common RAG Failure Modes and How to Fix Them

Part 11 of RAG in production · 1,362 words · ~6 min read · published 2026-08-24 · updated 2026-08-24 · Markdown variant

An operator runbook for the four RAG failure classes with no dedicated deep-dive elsewhere: retrieval miss, context overload, injection via content at ingestion time, and silent quality degradation — symptom, first diagnostic, and fix for each.

Part of the Agentic RAG in Production: The Complete Operator Guide guide.

The pillar guide names six recognizable failure classes in a field-guide table, each with a one-line symptom and diagnostic. Two of those six have a full runbook elsewhere in this cluster: a stale answer is covered by keeping a RAG index fresh, and a permission leak by permission-aware retrieval. This article expands the other four — retrieval miss, context overload, injection via content, and silent degradation — into symptom-diagnostic-fix runbooks with the mechanics the pillar's table cells only gesture at.

Why does a retriever return "no information found" when the answer exists?

A retrieval miss has three distinct root causes, and confusing them wastes an incident chasing the wrong fix. First, the chunk may never have entered the index — a parsing failure dropped a table, a crawl never reached the page, or a chunking boundary split the fact across two pieces so neither states it cleanly. Second, the chunk may be indexed but ranked below the cutoff the retriever applies, typically from an embedding-domain mismatch or a similarity threshold set too tight (RAG and retrieval for agents names this pairing directly). Third, the query may be an exact-identifier lookup — an error code, a SKU, a citation name — that vector-only search structurally misses, because semantic similarity was never designed to reward literal string matches the way a lexical index does.

The diagnostic sequence follows the same order:

1. Query the raw store directly for the expected passage, bypassing the ranker — a keyword filter settles whether the chunk is present at all.
2. If present, check where it ranks relative to the cutoff, and against what candidate-set size.
3. If the query names a literal identifier, check whether lexical search ran at all.

The fix maps to which of the three it turns out to be: an ingestion or chunking fix for a missing chunk, a swapped embedding model or a reranker for a threshold problem, and hybrid retrieval — lexical plus vector, fused — for the identifier case, since no embedding-quality tuning closes that gap alone.

Context overload: when the right passage is retrieved but the answer is still wrong

Context overload is a prompt-assembly failure, not a retrieval failure — the correct passage already made the candidate set, and the model still produced a wrong, hedged, or overly cautious answer because of how those candidates were packaged into the prompt. Three mechanics produce it:

• Lost-in-the-middle placement. A model reading a long assembled context under-weights content sitting between the start and end of the prompt (Liu et al., arXiv:2307.03172, TACL 2024, cited in RAG and retrieval for agents). Fix: sort the assembled context so the highest-scored chunks sit at the start or end.
• Chunk-count dilution. Feeding twenty marginally relevant chunks alongside the one correct chunk buries the signal in noise. Fix: a tighter candidate count after reranking, not a wider one.
• Contradictory chunks. Multiple versions of the same source disagree in the index, so the model hedges or blends a wrong average. Fix: deduplication at ingest and metadata-filtering by source recency.

The diagnostic step for all three is the same: pull the actual assembled prompt for a failing case, not just the retrieval log, and inspect chunk count, the position of the correct chunk, and whether any two chunks conflict. Retrieval logs alone hide this class entirely, because retrieval itself succeeded.

How does untrusted content end up in a RAG index in the first place?

Content injection reaches a RAG system at ingestion, before any agent or generation step touches a chunk. A document enters through a web crawl, a third-party connector (ticketing system, shared drive, partner feed), or a direct user upload, and none of those channels can be assumed free of instruction-like text — a support ticket can contain "ignore prior instructions," a scraped page can carry hidden text invisible to a human but plain to a parser. Once chunked and embedded, that text is indistinguishable from any other passage in the index unless something marked it untrusted on the way in.

Two ingestion-time controls address this, distinct from runtime guardrails applied when an agent later reads the content:

• Strip or escape instruction-like patterns before embedding. The same defense recommended for content entering a model's context applies earlier: "strip or escape instruction-like patterns (Ignore previous instructions, <system>, ASSISTANT:) from retrieved content before inserting into context" (agentic security checklist) — applied at chunk-creation time, a pattern is neutralized once for every future query that might retrieve it, instead of re-filtered on each read.
• Attribute every chunk to its source at ingest. Tag each chunk with the connector, document ID, and trust tier it came from, so a later incident becomes "connector X, document Y" to quarantine, rather than a corpus-wide search.

Sanitization at ingest is necessary but not sufficient alone: architectural patterns like Dual LLM and LLM Map-Reduce isolate untrusted content from tool-calling surfaces even after it clears ingestion, because "architecture reduces what an injection can reach; guardrails and checklists reduce how often one succeeds in the first place" (prompt injection design patterns). This runbook covers the ingestion layer; for what happens when retrieved content drives tool calls and downstream actions, see agentic RAG patterns.

Silent degradation: why RAG quality drifts without an obvious break

RAG quality degrades silently when one of three things changes without anyone watching, and none throws an application error — the system keeps answering, just with a slowly worsening hit rate. First, the embedding model or its provider changes behavior: a provider ships an update behind the same API identifier, and every query now scores against vectors produced by a slightly different function than the ones already indexed. Second, the corpus grows in ways that skew competition for top-k slots — a chunk once third out of five hundred candidates now ranks thirtieth out of fifty thousand, with nothing about the chunk itself having changed. Third, an ingestion connector breaks quietly — a schema change, an expired credential, a quota limit — and stops feeding new content while old content keeps answering fine, so the system looks healthy except for the growing gap of what it never learned.

The diagnostic for all three is a scheduled evaluation pass against a fixed golden set, run on a recurring cadence rather than only when a change ships, tracking recall@k and precision@k over time so a slow decline shows as a trend line instead of a complaint. This runbook does not re-derive how to build that harness — see evaluating a RAG system for golden-set and CI mechanics. Routing systems carry a narrower version of this same problem: a difficulty classifier can go stale after an upstream chunking or model change and misroute traffic between tiers unnoticed — see RAG cost and latency for that specific case.

What about stale answers and permission leaks?

Stale answers and permission leaks belong to this field guide but have their own dedicated runbooks, because their causes and fixes are architecturally distinct from the four above. A stale answer is a freshness problem, addressed by keeping a RAG index fresh with per-source staleness contracts and sync-lag monitoring. A permission leak is an authorization problem, addressed by permission-aware retrieval with pre-filters derived from the authenticated principal, applied before ranking. Treating either as a retrieval-quality or content-trust issue, as the four classes above are, misdirects the fix.

Turning this into an operator runbook

Two habits keep every failure class here fast to diagnose rather than slow to reconstruct after the fact. Log retrieval traces — the query, matched chunks with scores, and filters applied — with the same discipline as application logs, since every diagnostic step above depends on reconstructing what a query actually retrieved, not what it should have. And keep a standing incident table mapping each failure class to its owning team, diagnostic query, and typical fix, so the first responder to a wrong-answer page is not starting from a blank page. As of August 2026 that table is the cheapest artifact in this cluster to build — write it once, next to the pillar's production readiness checklist, and add a row whenever a new failure class earns one.

#rag #reliability #failure-modes #observability #agents #production