When and How to Rerank Retrieved Documents in RAG
Reranking as a budget decision: why first-stage ranking misorders good evidence, when cross-encoder reranking pays for itself, how to pick candidate depth at the knee, gating by query difficulty, and deduplicating after fusion.
- First-stage retrieval optimizes recall cheaply and accepts misordering; a reranker spends extra compute per candidate to fix the order. The production questions are budget questions — how many candidates reach the reranker, which queries deserve it, and whether the generator's context actually improves — not a yes-or-no adoption decision.
- Pick candidate depth at the measured knee of end-metric against depth: below it, the true best passage often never enters the candidate set; above it, latency and cost climb for negligible quality gain. Plot the curve once on your golden set and revisit when retrieval changes.
- Not every query deserves reranking. A cheap gate — score threshold or query-shape signal — sends easy lookups straight through and reserves rerank compute for ambiguous or high-stakes queries, where it moves outcomes most.
- Deduplicate near-identical passages after reranking and before assembly: fused results commonly contain the same content surfaced by both channels, and duplicated evidence overweights one idea in the generator's attention without adding information.
The pillar guide frames reranking as spending precision where it pays; this article makes the spending explicit. First-stage retrieval — lexical scoring, dense similarity, or their fusion — exists to get roughly-right candidates in milliseconds. Its scores are coarse: good enough to assemble a candidate set, not good enough to trust the order. The hybrid fusion article covers assembling that set; this one covers fixing its order.
Why is raw similarity ordering not good enough?
Three systematic misorderings show up again and again:
- Length bias: longer passages accumulate more term overlaps or higher similarity mass, so verbose near-misses outrank tight answers
- Lexical gaming: keyword-dense passages outscore genuinely explanatory ones that phrase things differently
- Fusion artifacts: merged lists carry positions influenced by two channels' quirks, not by evidence quality
A reranker — typically a cross-encoder reading query and passage jointly — judges each candidate as evidence for this question, which corrects all three at once. The price is that it must run per candidate at query time, which is why it lives late in the pipeline.
What does the depth knee look like?
Candidate depth trades two failure directions:
- Too shallow: the true best passage never enters the candidate list, so no amount of reranking skill recovers it
- Too deep: marginal candidates consume rerank compute and can crowd the final selection with mediocrity
Measure end-metric (answer quality proxy or recall@final-k after reranking) across depths on your golden set. The curve rises steeply, flattens at a knee, then crawls. Choose just past the knee — and log the date and retrieval configuration alongside the choice, because a chunking or embedding change shifts the curve.
When should reranking be gated per query?
Always-on reranking is simplest and correct at modest scale. Under volume, gate it:
| Signal | Route |
|---|---|
| Top fused score far above rest | skip — ordering already confident |
| Identifier-shaped exact match hit | skip — lexical already nailed it |
| Ambiguous phrasing, multi-intent | rerank |
| High-stakes tier (money, legal, security) | always rerank |
Every gate needs its own measurement: compare gated versus always-on on the golden set and on live feedback, and watch the miss cases the gate waves through. A router that saves 40% of rerank compute while quietly degrading 5% of hard answers has failed, whatever the dashboard says about cost (the corpus cost-and-latency reference covers the accounting).
How does reranking interact with context assembly?
Reranking optimizes ordering, but assembly decides what the generator actually sees, so the two stages need shared conventions:
- Selection size follows from the generator's context budget, not from reranker output count — a reranked list of fifty still gets truncated to the budget's worth
- Placement uses relevance: models attend unevenly across long contexts, so strongest evidence belongs early (and late), which requires exactly the scores reranking produces
- Attribution survives reranking: chunk identifiers carried through fusion must survive into the assembled prompt so answers can cite their evidence
- Score thresholds can replace fixed counts: when the tail of the reranked list scores far below the head, cutting it improves both cost and signal density
Teams that treat reranking as the last word on selection routinely re-dilute its gains at assembly time by stuffing everything back in. The reranker's output is a ranking plus confidence — let both drive assembly.
What happens to duplicates after fusion?
Hybrid fusion surfaces the same underlying text twice — once per channel — with different ranks. After reranking, collapse near-identical passages before prompt assembly: keep the highest-scoring representative, merge their source references, and free the slot for distinct evidence. Deduplication routinely buys more answer quality than another reranking stage would, at a fraction of the cost.
Near-identity itself needs a definition: exact-match dedup misses the common case where the two channels return the same passage with tiny extraction differences — different whitespace, one missing footnote. Practical implementations normalize aggressively (casefold, punctuation-strip, whitespace-collapse) and compare shingles or character n-grams rather than raw strings, with a similarity threshold tuned once on observed duplicates. Log every merge so threshold changes are auditable, and keep both source references on the survivor for attribution.
How do you know the reranker earns its keep?
Give the reranker the same accountability as everything else: golden-set comparison with it on versus off, reported separately for the query tiers your gate defines. Track rerank latency at p95 next to retrieval latency, cost per reranked query, and the fraction of queries where reranking changed the selected set. A reranker whose contribution metrics decay as first-stage quality improves has become a tax — retire or re-gate it. The reranking reference carries model-level mechanics; the operating rule stands alone here: rerankers are employees, and employees keep their jobs by measurable contribution.
One worked example makes the budget concrete. Suppose fusion returns 100 candidates at 40 milliseconds, and the reranker processes 50 of them in 120 milliseconds before assembly cuts to ten passages. If the golden set shows recall@10 rising from 0.71 to 0.86 with reranking on, and p95 latency still sits inside the product's budget, the spend is justified. If the same experiment shows recall moving 0.71 to 0.72 while p95 doubles, the honest conclusion is that first-stage ranking already orders this corpus well — and the compute belongs somewhere else. Both outcomes are wins; only skipping the measurement loses.
Frequently asked questions
- Why does first-stage retrieval misorder results if recall is high?
- First-stage scoring is built for speed over precision: lexical statistics and single-vector similarity both produce usable top-k lists quickly, but their scores are coarse signals that frequently place the best passage mid-list rather than first. If the generator sees candidates in the wrong order, attention favors whatever sits early regardless of true relevance. Reranking re-examines a small candidate set with an expensive-but-accurate model and repairs the ordering.
- How many candidates should go to the reranker?
- Enough that the best passage is usually present, few enough that cost stays flat: in practice teams plot end-task metric against candidate count on their golden set and choose the knee of that curve, commonly somewhere between 20 and 100 depending on corpus and traffic. Below the knee, quality is capped by recall into the reranker; above it, each additional candidate buys almost nothing while every millisecond shows up at p95.
- Should reranking run on every query?
- Usually no at scale. Simple lookups that first-stage ranking already ordered well waste rerank compute; ambiguous or high-stakes queries benefit most. A lightweight gate — a relevance-score threshold, a query-shape heuristic, or a small classifier — routes only the queries where reranking historically changed outcomes. Measure the gate itself against always-on reranking so its savings are real and its misses are known.
- What is the difference between a bi-encoder and a cross-encoder?
- A bi-encoder embeds query and document independently so vectors can be precomputed and searched fast — that is the first-stage design. A cross-encoder reads query and document together and outputs a relevance judgment, which is far more accurate per pair but requires running the pair through the model at query time. That cost asymmetry is exactly why cross-encoders belong late, on tens of pre-selected candidates, not early on thousands.
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.