Agentic RAG: Retrieval as a Tool for AI Agents
What agentic RAG is and how it differs from single-shot retrieval: tool contracts, iterative-retrieval budgets, multi-hop decomposition, and the trust boundary moving into the retrieval path.
Agentic RAG is retrieval-augmented generation where the model, not a fixed pipeline, decides whether to retrieve, writes its own queries, and iterates on what comes back — unlike single-shot RAG, which always retrieves once regardless of need. The retrieval engineering underneath doesn't change; what changes is who drives it.
Key facts
- Agentic RAG hands retrieval decisions to the model itself: it decides whether to retrieve, writes its own queries, reads results, and issues follow-ups, whereas single-shot RAG always retrieves once per query regardless of need.
- The standard integration pattern exposes retrieval as a single callable MCP tool — embed query, ANN search, rerank, return chunks — that the agent calls explicitly when it needs grounded context, enabling conditional and multi-hop retrieval.
- Multi-hop questions that depend on an intermediate entity ("who founded the company that acquired X?") fail as one dense query but succeed as a decomposed sequence where each hop's result informs the next.
- Iterative retrieval needs deterministic budgets — a cap on retrieval rounds, a token ceiling, and explicit stop conditions — since an unbounded "retrieve until satisfied" loop turns into an unbounded bill.
- Retrieved text becomes an observation that can trigger further tool calls, so agentic systems must treat it as untrusted input: strip instruction-like framing where feasible and carry chunk-level attribution through every step.
Tool contract, budgets, and trust
A retrieval tool exposed to an agent needs the same rigor as any tool definition: explicit parameters (query text, filters, result size), a result shape carrying slug, title, snippet, score, and stable URLs, and a description written for a literal-minded reader — the agent's only documentation. Give every task a budget of three numbers (rounds, tokens, stop conditions) and log agent-issued queries separately from human ones, since their failure modes differ.
The trust boundary moves too: in single-shot RAG, injected content in a chunk influences one answer; in agentic RAG the same text becomes an observation that can drive further tool calls and spend. Stripping instruction-like content, attributing every fact to its source chunk, and scoping which downstream tools an answering process may invoke move from optional hardening to a launch requirement.
For the full retrieval pipeline this sits on top of — chunking, hybrid search, reranking, evaluation — see /resources/rag-retrieval-for-agents. For tool-contract design and reliability patterns, see /resources/reliable-tool-calling. For the broader memory and context picture, see /resources/agent-memory-context.