How to Cut RAG Cost and Latency Without Cutting Quality
RAG cost and latency as engineered budgets: where the money actually goes, caching layers and their hit-rate economics, routing queries to right-sized models, bounding retrieval fan-out, the hidden lines (reindex migrations, eval compute), and p95 discipline.
- Every RAG query pays three bills: retrieval infrastructure, embedding inference, and generation tokens. Optimizing means attacking all three with measured levers — cache what repeats, route easy queries to cheaper models, and bound how deep each stage fans out.
- Caching decisions live or die on hit rate: response caches for repeated questions, semantic caches for near-duplicates, prompt-prefix caches for shared scaffolding. Measure hit rates before building elaborate invalidation machinery — a cache nobody hits is pure overhead.
- Route by difficulty, verified by evaluation: send easy queries to smaller generators and reserve premium models for hard tiers, but only after the golden set confirms small-model answers hold up on exactly the traffic you plan to route.
- Track p95 latency, not averages: tail latency is where users leave and timeouts cascade. Give every stage an explicit budget — first-stage retrieval, fusion, reranking, generation — and alarm on the stage that eats its neighbors' share.
The pillar guide lists cost and latency as the last demo-to-production delta; this article engineers it. The frame that keeps the work honest: every query pays three bills — retrieval infrastructure, embedding inference, generation tokens — and every optimization lever either reduces one of those bills or trades it against another. Nothing here requires cutting answer quality; everything requires measuring.
What should be cached first?
In order of payback:
- Prompt-prefix caching at the provider level: system prompts and tool schemas repeat every call; stable prefixes cut their cost dramatically (prompt caching). Structural requirement: volatile content goes last in the context.
- Response caching for identical questions within freshness tolerances — FAQ-shaped traffic often has heavy head (response caching).
- Semantic caching for paraphrase-level duplicates: powerful and risky; scope per tenant, threshold against the eval set, TTL to freshness bounds.
- Retrieval-result caching for identical query-plus-filter pairs, valid until corpus version changes.
Every layer's business case is its hit rate, measured in production before the invalidation complexity is built. A cache with a 3% hit rate is a liability wearing a discount sticker.
How does model routing lower the bill?
Generation is usually the dominant token bill, and not every question needs the flagship model. The pattern:
- Classify incoming queries into difficulty tiers (length, structure, prior outcomes)
- Verify with the golden set that the small model holds quality on the easy tier
- Route verified-easy traffic down; escalate on low confidence or high stakes
Two failure modes deserve standing alarms: silent quality drift after upstream changes (the router's assumptions expired) and classification flapping between tiers for borderline queries (users see inconsistent answer styles). Routing is cost-latency optimization's highest-leverage move and its easiest place to fool yourself.
Where does latency concentrate, and what bounds it?
A typical path stacks: first-stage retrieval (lexical + dense), fusion, reranking, prompt assembly, generation. Each needs an explicit millisecond budget, and the sum needs headroom below product expectations. The recurring offenders:
- Deep reranking — cap candidate depth at the measured knee (the reranking guide)
- Filtered vector search on selective filters — store-dependent; a poor fit punishes exactly your permission-heavy queries (vector database choice)
- Cold-start caches — semantic and response caches miss on first sight; warm them for known-hot content
- Token-count spikes — oversized assembled contexts slow generation; enforce assembly budgets rather than hoping
Set SLOs at p95 per stage and end to end, alarm stage-by-stage, and treat budget overruns as ownership assignments: the stage that ate its neighbor's milliseconds owns the fix.
How do you set the budgets themselves?
Budgets inherited from another system are guesses. Derive yours from three inputs: product expectations (what latency does the UX actually tolerate — interactive chat differs wildly from background synthesis), unit economics (what cost per answered query makes the product viable at target scale), and measured stage curves (how each stage's quality responds to spend — depth knees, cache thresholds, routing tiers). Write them as explicit numbers with owners, review quarterly against actuals, and re-derive whenever architecture shifts, because every upstream change in this cluster rewrites the curve underneath the budget.
Which costs hide outside the per-query view?
Five lines routinely escape per-query dashboards:
- Reindex migrations — full re-embeddings on model/chunking changes
- Evaluation compute — CI gates and scheduled suites are real spend, cheap relative to incidents
- Vector-store memory — scales with dimensionality × corpus × replicas
- Observability retention — retrieval traces accumulate fast
- Invalidation engineering — the human time caches and freshness contracts demand
Review these quarterly alongside the per-query unit economics (evaluation harnesses keep the quality side honest while you optimize).
What does disciplined look like?
Cost-per-query known and trending down at constant quality scores; cache hit rates published next to staleness bounds; routing rules version-controlled with their golden-set evidence; p95 latency budgets owned per stage; and the whole ledger reviewed when architecture changes — because every knob elsewhere in this cluster (chunking, embeddings, hybrid depth, rerankers, agentic loops) writes into this bill. Optimization without the instrument panel is just spending differently.
The quickest win sequence for a system that has never been costed: instrument per-stage latency and token counts for one week (the numbers are usually surprising); add prompt-prefix caching if the provider supports it (structural change only, no quality risk); then evaluate response caching on your query distribution. Those three steps typically move the bill double-digit percentages before any riskier lever — routing, semantic caches, depth cuts — enters the conversation.
And keep one number on the wall: quality at budget. Every optimization this article describes can be graded by the same golden set that gates correctness — caching by staleness-aware sampling, routing by tier-level scores, depth cuts by end-metric curves. When cost work and quality work share an instrument panel, they stop being opposing forces and become what they are operationally: two constraints of the same design problem.
If forced to choose a first month's agenda from everything above, it would be: instrument stages (week one), prefix caching plus response-cache evaluation (week two), reranking depth and candidate count re-derived against their curves (week three), routing pilot on the easy tier with shadow comparison (week four). Each step ships value alone, each produces numbers the next step consumes, and none requires touching quality-critical paths without a measured safety net. And when the month ends, the review artifact is one table — cost per query, p95 per stage, cache hit rates, quality scores — which is both the receipt for the work and the baseline for the next round.
Frequently asked questions
- Where does RAG cost actually concentrate?
- Usually generation tokens dominate at scale, followed by embedding inference for ingestion volume, then vector-store storage and memory. But the ranking shifts with architecture: aggressive reranking moves scoring cost up; frequent full re-embeddings move embedding cost up; high dimensionality moves storage up. Profile your own distribution before optimizing anything — the biggest line item varies an order of magnitude between systems.
- Is semantic caching safe to use?
- With guardrails: semantic caches return cached answers for near-duplicate questions, so a too-loose similarity threshold serves stale or subtly wrong answers to questions that differ meaningfully. Scope caches per tenant (never cross permission boundaries), set conservative thresholds validated against your eval set, attach TTLs aligned to corpus freshness, and monitor disagreement cases where users rephrase because the first answer missed.
- How does model routing work without hurting quality?
- Define difficulty tiers for incoming queries, verify on your golden set that smaller generators match large-model quality on the easy tier, then route only verified-easy traffic downward. Keep a shadow comparison running after rollout: if easy-tier answers degrade after a corpus or chunking change, the router silently became wrong. Routing is a hypothesis under continuous test, not a configuration.
- What are the commonly forgotten RAG costs?
- Reindex migrations (full re-embeddings when changing models or chunking), evaluation-suite compute running continuously in CI, vector-store memory which scales with dimensionality and traffic, observability storage for retrieval traces, and the engineering time invalidation complexity demands. Budget reviews that count only per-query inference miss most of the real bill.
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.