{
  "slug": "rag-retrieval-for-agents",
  "title": "RAG and Retrieval for Agents",
  "description": "End-to-end practitioner reference for Retrieval-Augmented Generation: pipeline stages, chunking strategies, dense/sparse/hybrid retrieval, reranking, agentic retrieval patterns, quality failure modes, and evaluation — with verified sources for every named technique.",
  "category": "Guide",
  "tags": [
    "rag",
    "retrieval",
    "embeddings",
    "chunking",
    "reranking",
    "agents",
    "vector-databases",
    "evaluation"
  ],
  "updated": "2026-08-05",
  "premium": false,
  "canonical": "https://changegamer.ai/resources/rag-retrieval-for-agents",
  "markdown": "https://changegamer.ai/resources/rag-retrieval-for-agents.md",
  "outline": [
    {
      "depth": 2,
      "text": "Key facts",
      "anchor": "key-facts"
    },
    {
      "depth": 2,
      "text": "Pipeline stages",
      "anchor": "pipeline-stages"
    },
    {
      "depth": 2,
      "text": "Chunking strategies",
      "anchor": "chunking-strategies"
    },
    {
      "depth": 2,
      "text": "Retrieval methods",
      "anchor": "retrieval-methods"
    },
    {
      "depth": 2,
      "text": "Reranking",
      "anchor": "reranking"
    },
    {
      "depth": 2,
      "text": "Agentic retrieval patterns",
      "anchor": "agentic-retrieval-patterns"
    },
    {
      "depth": 2,
      "text": "Quality and failure modes",
      "anchor": "quality-and-failure-modes"
    },
    {
      "depth": 2,
      "text": "Evaluation",
      "anchor": "evaluation"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "reranking-for-rag",
      "title": "Reranking for RAG: Cross-Encoders, LLM Rerankers, and Hosted APIs",
      "description": "Second-stage retrieval step that re-scores bi-encoder candidates with full query-document attention, boosting precision without sacrificing recall; covers cross-encoder, LLM, and late-interaction reranking, hosted APIs, sizing heuristics, and evaluation.",
      "url": "https://changegamer.ai/resources/reranking-for-rag"
    },
    {
      "slug": "chunking-strategies-for-rag",
      "title": "Chunking Strategies for RAG",
      "description": "Practitioner reference for chunking documents before embedding: fixed-size, recursive, semantic, late chunking, and contextual retrieval — with a strategy comparison table, chunk-size and overlap tradeoffs, code/table/Markdown handling, embedding model context limits, and evaluation methods.",
      "url": "https://changegamer.ai/resources/chunking-strategies-for-rag"
    },
    {
      "slug": "choosing-a-vector-database",
      "title": "Choosing a Vector Database",
      "description": "Criteria-based decision guide: dedicated vs. add-on vector stores, scale thresholds, hybrid search support, self-host vs. managed, and a start-here recommendation.",
      "url": "https://changegamer.ai/resources/choosing-a-vector-database"
    },
    {
      "slug": "hybrid-search-for-rag",
      "title": "Hybrid Search for RAG: BM25 + Dense Retrieval and Fusion",
      "description": "How to combine lexical (BM25/SPLADE) and dense vector retrieval with Reciprocal Rank Fusion for higher first-stage recall in RAG pipelines — with the RRF formula, a sparse-method comparison table, and verified DB support.",
      "url": "https://changegamer.ai/resources/hybrid-search-for-rag"
    }
  ],
  "furtherReading": [
    {
      "slug": "acp-vs-ap2-vs-x402",
      "title": "ACP vs. AP2 vs. x402: Which Agent Payment Rail Should You Implement?",
      "description": "A decision framework for choosing between ACP, AP2, and x402 (plus the self-hosted 402 gate) — sorted by who your buyer actually is, what you are selling, and what is live versus waitlisted today.",
      "url": "https://changegamer.ai/articles/acp-vs-ap2-vs-x402"
    },
    {
      "slug": "agent-checkout-vs-human-checkout",
      "title": "Agent Checkout vs. Human Checkout: Why Your Payment Flow Fails Machine Buyers",
      "description": "Why checkout built for a person watching a screen is unusable by an AI agent, and what a checkout flow that actually completes for a machine buyer looks like — 402 + API key versus native x402.",
      "url": "https://changegamer.ai/articles/agent-checkout-vs-human-checkout"
    }
  ],
  "body": "Retrieval quality is the primary bottleneck for grounded agents. A model with perfect reasoning over wrong or missing context still produces wrong answers. This guide synthesizes the techniques agent builders actually need — concepts first, hype omitted.\n\n## Key facts\n\n- Retrieval quality is the real bottleneck for grounded agents — even a model with flawless reasoning produces wrong answers when the context handed to it is wrong or missing.\n- The pipeline runs through a fixed sequence — parse, chunk, embed, index, retrieve, optionally rerank, assemble, generate — and each stage introduces its own characteristic failure, from garbled parsing to hallucination despite grounding.\n- Chunking ranges from simple fixed-size splitting to structure-aware and semantic methods, up through newer techniques like late chunking and Anthropic's contextual retrieval, the latter reportedly cutting failed lookups by roughly half on its own and further once paired with a reranker.\n- Combining dense (embedding/ANN) and sparse (BM25) retrieval through rank-based fusion such as RRF covers both semantic and exact-keyword matching better than relying on either method by itself.\n- Because approximate nearest-neighbor search only estimates relevance, a cross-encoder reranker is commonly layered on top: pull a cheap, wide candidate set first, then rerank it down to a small, precise one.\n- Agentic patterns hand retrieval decisions to the model itself — rewriting or decomposing queries, chaining multi-hop lookups, exposing retrieval as a callable tool, and verifying retrieved context before treating it as ground truth.\n\n## Pipeline stages\n\nA RAG pipeline has a fixed skeleton. Each stage has distinct failure modes.\n\n| Stage | What happens | Failure mode |\n|---|---|---|\n| **Ingest / parse** | Documents loaded and converted to plain text | Garbled tables, lost structure, wrong encoding |\n| **Chunk** | Text split into retrieval units | Boundaries break mid-concept; size/overlap wrong for query patterns |\n| **Embed** | Chunks encoded to dense vectors | Domain mismatch degrades recall; stale model on new vocabulary |\n| **Index** | Vectors stored in ANN index; optional BM25 index built in parallel | Index staleness; missing incremental update path |\n| **Retrieve** | Query embedded; top-k fetched by ANN similarity or lexical score | Irrelevant chunks pass threshold; relevant chunks below cutoff |\n| **Rerank** (optional) | Cross-encoder scores and reorders top-k | Reranker latency budget exceeded; reranker domain mismatch |\n| **Assemble context** | Top chunks formatted and injected into prompt | Lost-in-the-middle; token budget overflow; contradictory chunks |\n| **Generate** | LLM produces answer grounded in context | Hallucination despite context; faithful-but-wrong summary |\n\n## Chunking strategies\n\n**Fixed-size + overlap** — split every N tokens with an M-token overlap between adjacent chunks. Simple and fast. Overlap prevents boundary-straddling facts from disappearing, but does not guarantee semantic coherence at cut points. Most frameworks default to ~512 tokens with ~50–100 token overlap.\n\n**Recursive / structural splitting** — split on document structure in priority order (section headings → paragraphs → sentences → characters). Preserves logical boundaries. Preferred over fixed-size when documents have reliable structure (Markdown, HTML, code).\n\n**Semantic chunking** — embed consecutive sentences; measure cosine similarity between adjacent embeddings; start a new chunk when similarity drops below a threshold. Groups semantically coherent content regardless of character count. Higher ingestion cost than fixed-size; similarity threshold requires tuning.\n\n**Late chunking** (Jina AI, arXiv:2409.04701) — embed the full document first using a long-context embedding model, then chunk the resulting token embeddings via pooling. Each chunk embedding captures the full document context rather than only local context. Works without retraining; requires a long-context embedding model.\n\n**Contextual retrieval** (Anthropic, September 2024) — prepend a short LLM-generated context summary (typically 50–100 tokens) to each chunk before embedding and before building the BM25 index. The summary situates the chunk within the source document. Anthropic reported up to 49% reduction in failed retrievals; up to 67% when combined with reranking. Uses prompt caching to keep per-chunk generation cost low.\n\n**Chunk size and boundary choice** matter because retrieval is tuned for a size distribution: chunks too large dilute signal; chunks too small lose context. Boundary placement determines whether a concept spans two chunks (bad) or sits within one (good). There is no universal optimum — calibrate against your query distribution.\n\n## Retrieval methods\n\n**Dense retrieval (embeddings + ANN)** — embed the query; find the nearest chunk vectors in an approximate nearest neighbor (ANN) index (typically HNSW). Fast at scale; captures semantic similarity even when exact query words are absent. Quality depends on the embedding model's domain fit.\n\n**Sparse / lexical retrieval (BM25)** — rank chunks by term frequency and inverse document frequency using the Okapi BM25 function (Robertson & Spärck Jones, 1970s–1990s). No embedding required; exact keyword matches score high. Fails for paraphrases or terminology gaps between query and document.\n\n**Hybrid search** — run dense and sparse retrieval in parallel; merge ranked lists. Covers both semantic and exact-match signals. The standard fusion algorithm is Reciprocal Rank Fusion (RRF, Cormack et al., SIGIR 2009): each document's score is the sum of 1/(k + rank) across ranked lists (k=60 is the conventional default). RRF ignores raw scores and works on ranks only, sidestepping the score-normalization problem that makes direct score combination fragile. Elasticsearch and OpenSearch both ship native RRF retrievers.\n\n## Reranking\n\nANN retrieval is approximate by design. A cross-encoder reranker takes the top-k results and scores each query–chunk pair with full joint attention — much more accurate than embedding similarity but too slow for the full index. The workflow: retrieve top-50 or top-100 cheaply, then rerank to top-10 expensively.\n\nVerified rerankers available as APIs or open weights:\n\n- **Cohere Rerank** (API) — `rerank-v4.0-pro` and `rerank-v4.0-fast`; supports structured data formatted as YAML strings. Docs: docs.cohere.com/docs/rerank-overview.\n- **BGE reranker** (BAAI / open-weight) — `BAAI/bge-reranker-v2-m3` (multilingual, M3 backbone); `BAAI/bge-reranker-v2.5-gemma2-lightweight` (token compression for efficiency). Available on Hugging Face; used via the FlagEmbedding library.\n- **Jina reranker** (Jina AI / API + open-weight) — `jina-reranker-v3.5` is reportedly the current API model as of 2026-08-05 (a drop-in upgrade superseding `jina-reranker-v3`, same request schema; WebSearch-corroborated across its arXiv paper and model card — not independently re-fetched, Jina's own site 403'd this cycle); `jinaai/jina-reranker-v2-base-multilingual` is the open-weight cross-encoder (100+ languages, 1 024-token context, CC-BY-NC 4.0). Docs: jina.ai/reranker.\n\n## Agentic retrieval patterns\n\n**Query rewriting / decomposition** — before retrieval, use an LLM to rewrite the user query into a better retrieval query, or decompose a complex question into multiple sub-queries, each retrieved independently. Addresses vocabulary mismatch and multi-part questions that no single chunk answers.\n\n**Multi-hop retrieval** — answer a chain of retrieval steps where each hop's result informs the next query. Required when the answer depends on facts that are only linked via an intermediate entity (\"Who founded the company that acquired X?\" → retrieve X's acquirer → retrieve acquirer's founder).\n\n**Retrieval as an MCP tool** — expose the full RAG pipeline (embed query → ANN search → rerank → return chunks) as a single MCP tool. The agent calls it explicitly when it needs grounded context rather than having retrieval injected automatically. This is the \"RAG-as-a-tool\" pattern: the agent decides *when* to retrieve and with *what* query, enabling conditional retrieval and multi-hop chains. Contrast with classic single-shot RAG where retrieval is always triggered before generation.\n\n**Self-correction** — after retrieval, have the agent (or a separate verification step) check whether the retrieved context actually supports the planned answer before generating it. If context is insufficient or contradictory, re-query with a refined query or surface uncertainty explicitly. Treat retrieved content as untrusted input, not ground truth (see /resources/agentic-security-checklist for context poisoning risks).\n\nSee /resources/agent-memory-context for how RAG-based semantic memory relates to other memory types in agents.\n\n## Quality and failure modes\n\n| Failure mode | Cause | Mitigation |\n|---|---|---|\n| Irrelevant chunks | Embedding domain mismatch; threshold too loose | Fine-tune or swap embedding model; tighten top-k cutoff; add reranker |\n| Contradictory chunks | Multiple source versions in index | Dedup at ingest; metadata-filter by source recency; surface contradictions explicitly |\n| Lost-in-the-middle | Critical chunk placed in middle of long context | Place highest-scored chunks at start/end; see Liu et al. arXiv:2307.03172 (TACL 2024) — also covered in /resources/agent-memory-context |\n| Stale index | Source documents updated after ingestion | Incremental re-ingestion pipeline; TTL-based invalidation; `updated` metadata on chunks |\n| Embedding-domain mismatch | General-purpose embedder on specialized domain | Domain-adaptive fine-tuning; switch to a specialized embedding model |\n| Context poisoning | Adversarial content in the retrieval corpus | Validate chunks before injection; treat retrieved text as untrusted data (see /resources/agentic-security-checklist) |\n\nFor reliable structured outputs from the generation step, see /resources/reliable-tool-calling.\n\n## Evaluation\n\n**Retrieval metrics** — precision@k (fraction of retrieved chunks that are relevant); recall@k (fraction of relevant chunks retrieved). Measure both: high precision with low recall means you miss facts; high recall with low precision floods the context window.\n\n**Generation metrics** — faithfulness / groundedness (does the answer follow from the retrieved context, with no hallucinated claims?); answer relevance (does the answer address the question?). These require either human judges or LLM-as-judge scoring.\n\n**RAGAS** (github.com/vibrantlabsai/ragas — the project rebranded from explodinggradients to Vibrant Labs in 2026; old org links redirect) is the standard open-source framework for reference-free RAG evaluation. It computes faithfulness, answer relevance, and context precision/recall without requiring ground-truth annotations, and integrates with LangChain and LlamaIndex. Paper: arXiv:2309.15217.\n\n## Verified sources\n\n- Late Chunking paper (Jina AI / Weaviate, arXiv:2409.04701): https://arxiv.org/abs/2409.04701\n- Late Chunking GitHub (Jina AI): https://github.com/jina-ai/late-chunking\n- Contextual Retrieval (Anthropic engineering blog, September 2024): https://www.anthropic.com/engineering/contextual-retrieval\n- Reciprocal Rank Fusion — hybrid retrieval analysis (arXiv:2210.11934): https://arxiv.org/abs/2210.11934\n- RRF in OpenSearch: https://opensearch.org/blog/introducing-reciprocal-rank-fusion-hybrid-search/\n- ColBERT — late interaction retrieval (arXiv:2004.12832, SIGIR 2020): https://arxiv.org/abs/2004.12832\n- Cohere Rerank overview: https://docs.cohere.com/docs/rerank-overview\n- Cohere Rerank API reference: https://docs.cohere.com/reference/rerank\n- BGE reranker-v2-m3 (BAAI, Hugging Face): https://huggingface.co/BAAI/bge-reranker-v2-m3\n- BGE reranker docs: https://bge-model.com/bge/bge_reranker_v2.html\n- Jina reranker API: https://jina.ai/reranker/\n- jina-reranker-v3 (Jina AI, superseded by v3.5 — kept for historical reference): https://jina.ai/models/jina-reranker-v3/\n- jina-reranker-v2-base-multilingual (Hugging Face): https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual\n- jina-reranker-v3.5 model card (Jina AI, WebSearch-corroborated as of 2026-08-05): https://jina.ai/models/jina-reranker-v3.5/\n- jina-reranker-v3.5 paper (arXiv:2607.18152, WebSearch-corroborated as of 2026-08-05): https://arxiv.org/abs/2607.18152\n- RAGAS framework (GitHub, vibrantlabsai; formerly explodinggradients): https://github.com/vibrantlabsai/ragas\n- RAGAS paper (arXiv:2309.15217, EACL 2024): https://aclanthology.org/2024.eacl-demo.16/\n- Okapi BM25 (Wikipedia reference): https://en.wikipedia.org/wiki/Okapi_BM25\n- \"Lost in the Middle\" — context position effects (Liu et al., TACL 2024, arXiv:2307.03172): https://arxiv.org/abs/2307.03172",
  "sources": [
    "https://arxiv.org/abs/2409.04701",
    "https://github.com/jina-ai/late-chunking",
    "https://www.anthropic.com/engineering/contextual-retrieval",
    "https://arxiv.org/abs/2210.11934",
    "https://opensearch.org/blog/introducing-reciprocal-rank-fusion-hybrid-search/",
    "https://arxiv.org/abs/2004.12832",
    "https://docs.cohere.com/docs/rerank-overview",
    "https://docs.cohere.com/reference/rerank",
    "https://huggingface.co/BAAI/bge-reranker-v2-m3",
    "https://bge-model.com/bge/bge_reranker_v2.html",
    "https://jina.ai/reranker/",
    "https://jina.ai/models/jina-reranker-v3/",
    "https://huggingface.co/jinaai/jina-reranker-v2-base-multilingual",
    "https://jina.ai/models/jina-reranker-v3.5/",
    "https://arxiv.org/abs/2607.18152",
    "https://github.com/vibrantlabsai/ragas",
    "https://aclanthology.org/2024.eacl-demo.16/",
    "https://en.wikipedia.org/wiki/Okapi_BM25",
    "https://arxiv.org/abs/2307.03172"
  ]
}