How to Evaluate a RAG System (Retrieval Metrics, Generation Metrics, CI Gates)
The evaluation harness that keeps RAG changeable: golden-set construction, retrieval metrics separated from generation metrics, LLM-as-judge screening with human acceptance, CI regression gates, and the logged-query flywheel.
- A RAG system without evaluation cannot be safely changed — every knob (chunking, embeddings, fusion, reranking, prompts) moves quality in ways nobody can eyeball. The harness converts those knobs from acts of faith into measured engineering decisions.
- Build a version-controlled golden set: real questions annotated with the passages that must be retrieved and reference answers. Cover exact-identifier lookups, paraphrases, multi-hop questions, empty-result cases, and permission-negative cases that must return nothing for some users.
- Score retrieval separately from generation. Recall@k, MRR and nDCG are deterministic, fast and model-free — ideal CI gates. Faithfulness and answer relevance are noisier generation-side measures; use judges for regression screening and humans for acceptance.
- Wire the suite into CI so any change to chunking, models, prompts or retrieval parameters runs it automatically, and treat threshold breaches as build failures. Then close the loop with production query logs: sample failures, cluster them into intents, promote the instructive ones into the golden set.
Everything else in this cluster (ingestion, chunking, embeddings, fusion, reranking) produces knobs that move answer quality invisibly. Evaluation is the instrument panel that makes those knobs safe to turn — and per the pillar guide, it is what separates maintainable systems from demos that decay.
What goes into a golden set?
Five question classes, drawn from real traffic rather than imagination:
- Exact-identifier lookups — codes, names, part numbers, citations
- Paraphrases — questions whose wording differs from the source documents
- Multi-hop questions — answers requiring evidence from more than one passage
- Empty-result cases — questions the corpus genuinely cannot answer, where the correct behavior is a confident "not found"
- Permission-negative cases — questions answerable only from content a given principal must not see
Annotate each with its required passages and, where practical, reference answers. Version the set in git next to the code it tests; evolve it through review, not by silent edits. Thirty disciplined cases outperform three thousand sloppy ones because every case earns its place by representing something real.
Which metrics gate which layer?
Keep the layers strictly separate:
| Layer | Metric | Property |
|---|---|---|
| Retrieval | recall@k | did needed passage enter top-k |
| Retrieval | MRR / nDCG | how high did it rank |
| Generation | faithfulness | is the answer entailed by retrieved context |
| Generation | relevance | does it address the asked question |
Retrieval metrics are deterministic, fast, and model-independent — they belong in every pull request's checks. Generation metrics need either rubric-scored human review or LLM-as-judge patterns; judges are cheap and noisy, humans are ground truth and expensive, so screen with judges continuously and calibrate them against humans on a sample, reserving people for acceptance and disagreements.
What makes judge-based evaluation drift?
Judges are language models, so they inherit every instability of language models:
- Prompt sensitivity: rewording the rubric shifts scores systematically, which reads as a quality change when it is a measurement change
- Model updates: swapping or upgrading the judge silently re-bases your entire metric history; pin the judge model and version it like any other dependency
- Position and verbosity bias: judges favor longer answers and sometimes earlier options; randomize presentation order where applicable and cap answer lengths before judging
- Self-preference: a judge from the same family as the generator grades that generator generously
The countermeasures are procedural: calibrate against human labels on a sample each cycle, track judge-versus-human agreement as its own metric, and route disagreement cases to humans rather than averaging them away.
How do you keep the golden set honest?
Golden sets rot in three ways: they leak into prompts (someone pastes test questions into few-shot examples), they overfit through repeated tuning against them, and they stop representing traffic as the product evolves. Countermeasures: hold out a rotation slice never used during development; refresh question phrasings periodically while keeping annotations stable; review additions through the same pull-request discipline as code; and maintain a strict no-training-on-the-golden-set rule for prompts, retrievers and routers alike.
How does evaluation wire into delivery?
Two loops with different clocks:
- Per-change loop (CI): on every PR touching chunking, models, prompts or retrieval parameters, rebuild a sample index, run the offline suite, fail the build beyond thresholds. This is the regression gate that makes bold changes boring to ship
- Scheduled loop: a periodic full pass over current production data catches environmental drift — corpus growth skewing rankings, connector changes shifting text distributions — that per-change runs on stable samples cannot see
Treat gate thresholds as reviewed configuration with history, not folklore. When a metric improves, tighten thoughtfully; when it regresses, the fix is reverting or fixing the change, never quietly lowering the bar (the deploy gates article pattern applies here too).
How does the harness improve after launch?
Through the logged-query flywheel. Sample production queries on a schedule, cluster them by intent, and inspect the failures — questions with no clicks, negative feedback, or escalations. Promote instructive failures into the golden set with annotations, so the suite grows toward the system's real weak spots rather than its imagined ones. The same logs surface coverage gaps: an intent cluster with consistently poor retrieval is next quarter's ingestion or chunking work, discovered by measurement instead of complaint. Observability feeds the flywheel — agent observability covers the instrumentation side; the evaluation side closes it by turning telemetry back into tests.
The result compounds: every incident becomes a fixture, every fixture sharpens every future decision, and the team stops experiencing RAG as fragile — because nothing changed without the instrument panel noticing first.
A closing calibration on effort. The harness described here costs real time to build — typically days for the first golden set and gate, not weeks — and pays it back at the first significant change it screens: one prevented bad deploy of a chunking migration, one caught prompt regression, one embedding switch that shadow-testing rejected, repays the build several times over. Teams skip evaluation because everything works at launch. That is precisely when the harness is cheapest to build and the absence of it most expensive later, since every knob added afterward multiplies the number of silent ways the system can regress.
Start small if starting is hard: ten golden questions, one retrieval metric, one CI step. The harness's value compounds from existing, not from being complete on day one.
Frequently asked questions
- What is a golden set for RAG evaluation?
- A curated, version-controlled set of representative questions with annotations: which passages must be retrieved for each question, plus reference answers. Quality beats quantity — thirty well-chosen cases spanning your real query distribution catch more regressions than thousands assembled carelessly. It should include negative cases (questions the system must decline or return empty) and permission cases (queries that must yield nothing for unauthorized users).
- Which metrics measure retrieval quality?
- Recall@k asks whether a needed passage made the top-k candidates; MRR and nDCG score how high it ranked. All three are deterministic, fast, and independent of any language model, which makes them ideal for continuous integration gates. Most "answers got worse" reports resolve here: generation did not degrade, retrieval stopped surfacing the evidence the generator needed.
- Can an LLM judge evaluate RAG answers reliably?
- Reliably enough for regression screening when calibrated: give the judge the retrieved context and ask whether the answer is faithful to it, plus whether it addresses the question. Judge scores are noisy at fine granularity, drift with prompt changes, and inherit model biases — so treat them as cheap screening, validate them periodically against human judgments, and reserve humans for acceptance decisions and the disagreement cases.
- How often should a RAG evaluation run?
- On every proposed change in CI — chunking, embedding model, fusion weights, reranker depth, prompts — because each of those knobs shifts answer quality invisibly. Additionally run a scheduled full pass against production data weekly or monthly to catch environmental drift (corpus growth, query-mix shifts) that per-change runs cannot see. A red gate blocks merge exactly like a failing unit test.
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.