# When and How to Rerank Retrieved Documents in RAG

> Reranking as a budget decision: why first-stage ranking misorders good evidence, when cross-encoder reranking pays for itself, how to pick candidate depth at the knee, gating by query difficulty, and deduplicating after fusion.

Guide: RAG in production — part 5
Published: 2026-08-23 · Updated: 2026-08-23 · 903 words · ~1201 tokens (estimate)
Canonical: https://changegamer.ai/articles/reranking-retrieved-results
JSON: https://changegamer.ai/api/articles/reranking-retrieved-results.json
Pillar: https://changegamer.ai/articles/rag-in-production.md

## In short

- First-stage retrieval optimizes recall cheaply and accepts misordering; a reranker spends extra compute per candidate to fix the order. The production questions are budget questions — how many candidates reach the reranker, which queries deserve it, and whether the generator's context actually improves — not a yes-or-no adoption decision.
- Pick candidate depth at the measured knee of end-metric against depth: below it, the true best passage often never enters the candidate set; above it, latency and cost climb for negligible quality gain. Plot the curve once on your golden set and revisit when retrieval changes.
- Not every query deserves reranking. A cheap gate — score threshold or query-shape signal — sends easy lookups straight through and reserves rerank compute for ambiguous or high-stakes queries, where it moves outcomes most.
- Deduplicate near-identical passages after reranking and before assembly: fused results commonly contain the same content surfaced by both channels, and duplicated evidence overweights one idea in the generator's attention without adding information.

---


The [pillar guide](/articles/rag-in-production) frames reranking as spending precision where it pays; this article makes the spending explicit. First-stage retrieval — lexical scoring, dense similarity, or their fusion — exists to get roughly-right candidates in milliseconds. Its scores are coarse: good enough to assemble a candidate set, not good enough to trust the order. The [hybrid fusion article](/articles/hybrid-retrieval-fusion) covers assembling that set; this one covers fixing its order.

## Why is raw similarity ordering not good enough?

Three systematic misorderings show up again and again:

1. **Length bias**: longer passages accumulate more term overlaps or higher similarity mass, so verbose near-misses outrank tight answers
2. **Lexical gaming**: keyword-dense passages outscore genuinely explanatory ones that phrase things differently
3. **Fusion artifacts**: merged lists carry positions influenced by two channels' quirks, not by evidence quality

A reranker — typically a cross-encoder reading query and passage jointly — judges each candidate as evidence for this question, which corrects all three at once. The price is that it must run per candidate at query time, which is why it lives late in the pipeline.

## What does the depth knee look like?

Candidate depth trades two failure directions:

- Too shallow: the true best passage never enters the candidate list, so no amount of reranking skill recovers it
- Too deep: marginal candidates consume rerank compute and can crowd the final selection with mediocrity

Measure end-metric (answer quality proxy or recall@final-k after reranking) across depths on your golden set. The curve rises steeply, flattens at a knee, then crawls. Choose just past the knee — and log the date and retrieval configuration alongside the choice, because a chunking or embedding change shifts the curve.

## When should reranking be gated per query?

Always-on reranking is simplest and correct at modest scale. Under volume, gate it:

| Signal | Route |
|---|---|
| Top fused score far above rest | skip — ordering already confident |
| Identifier-shaped exact match hit | skip — lexical already nailed it |
| Ambiguous phrasing, multi-intent | rerank |
| High-stakes tier (money, legal, security) | always rerank |

Every gate needs its own measurement: compare gated versus always-on on the golden set and on live feedback, and watch the miss cases the gate waves through. A router that saves 40% of rerank compute while quietly degrading 5% of hard answers has failed, whatever the dashboard says about cost (the corpus cost-and-latency reference covers the accounting).

## How does reranking interact with context assembly?

Reranking optimizes ordering, but assembly decides what the generator actually sees, so the two stages need shared conventions:

- **Selection size** follows from the generator's context budget, not from reranker output count — a reranked list of fifty still gets truncated to the budget's worth
- **Placement uses relevance**: models attend unevenly across long contexts, so strongest evidence belongs early (and late), which requires exactly the scores reranking produces
- **Attribution survives reranking**: chunk identifiers carried through fusion must survive into the assembled prompt so answers can cite their evidence
- **Score thresholds can replace fixed counts**: when the tail of the reranked list scores far below the head, cutting it improves both cost and signal density

Teams that treat reranking as the last word on selection routinely re-dilute its gains at assembly time by stuffing everything back in. The reranker's output is a ranking plus confidence — let both drive assembly.

## What happens to duplicates after fusion?

Hybrid fusion surfaces the same underlying text twice — once per channel — with different ranks. After reranking, collapse near-identical passages before prompt assembly: keep the highest-scoring representative, merge their source references, and free the slot for distinct evidence. Deduplication routinely buys more answer quality than another reranking stage would, at a fraction of the cost.

Near-identity itself needs a definition: exact-match dedup misses the common case where the two channels return the same passage with tiny extraction differences — different whitespace, one missing footnote. Practical implementations normalize aggressively (casefold, punctuation-strip, whitespace-collapse) and compare shingles or character n-grams rather than raw strings, with a similarity threshold tuned once on observed duplicates. Log every merge so threshold changes are auditable, and keep both source references on the survivor for attribution.

## How do you know the reranker earns its keep?

Give the reranker the same accountability as everything else: golden-set comparison with it on versus off, reported separately for the query tiers your gate defines. Track rerank latency at p95 next to retrieval latency, cost per reranked query, and the fraction of queries where reranking changed the selected set. A reranker whose contribution metrics decay as first-stage quality improves has become a tax — retire or re-gate it. The [reranking reference](/resources/reranking-for-rag) carries model-level mechanics; the operating rule stands alone here: rerankers are employees, and employees keep their jobs by measurable contribution.

One worked example makes the budget concrete. Suppose fusion returns 100 candidates at 40 milliseconds, and the reranker processes 50 of them in 120 milliseconds before assembly cuts to ten passages. If the golden set shows recall@10 rising from 0.71 to 0.86 with reranking on, and p95 latency still sits inside the product's budget, the spend is justified. If the same experiment shows recall moving 0.71 to 0.72 while p95 doubles, the honest conclusion is that first-stage ranking already orders this corpus well — and the compute belongs somewhere else. Both outcomes are wins; only skipping the measurement loses.


## Frequently asked questions

### Why does first-stage retrieval misorder results if recall is high?

First-stage scoring is built for speed over precision: lexical statistics and single-vector similarity both produce usable top-k lists quickly, but their scores are coarse signals that frequently place the best passage mid-list rather than first. If the generator sees candidates in the wrong order, attention favors whatever sits early regardless of true relevance. Reranking re-examines a small candidate set with an expensive-but-accurate model and repairs the ordering.

### How many candidates should go to the reranker?

Enough that the best passage is usually present, few enough that cost stays flat: in practice teams plot end-task metric against candidate count on their golden set and choose the knee of that curve, commonly somewhere between 20 and 100 depending on corpus and traffic. Below the knee, quality is capped by recall into the reranker; above it, each additional candidate buys almost nothing while every millisecond shows up at p95.

### Should reranking run on every query?

Usually no at scale. Simple lookups that first-stage ranking already ordered well waste rerank compute; ambiguous or high-stakes queries benefit most. A lightweight gate — a relevance-score threshold, a query-shape heuristic, or a small classifier — routes only the queries where reranking historically changed outcomes. Measure the gate itself against always-on reranking so its savings are real and its misses are known.

### What is the difference between a bi-encoder and a cross-encoder?

A bi-encoder embeds query and document independently so vectors can be precomputed and searched fast — that is the first-stage design. A cross-encoder reads query and document together and outputs a relevance judgment, which is far more accurate per pair but requires running the pair through the model at query time. That cost asymmetry is exactly why cross-encoders belong late, on tens of pre-selected candidates, not early on thousands.


---

## The rest of this guide

- [Agentic RAG in Production: The Complete Operator Guide](https://changegamer.ai/articles/rag-in-production.md): The operator playbook for agentic RAG in production: ingestion, chunking, hybrid retrieval, reranking, evaluation, freshness and cost.
- [How to Build a RAG Ingestion Pipeline That Survives Production](https://changegamer.ai/articles/rag-ingestion-pipeline.md): The six properties that separate a production RAG ingestion pipeline from a demo script: tested extraction, idempotent writes, incremental updates, deletion propagation, durable execution, and metadata captured at ingest time.
- [How to Chunk Documents for RAG (Strategy Beats Size)](https://changegamer.ai/articles/chunking-documents-for-rag.md): Chunking decisions that actually move retrieval quality: structural boundaries before fixed windows, parent-document expansion, special handling for tables and code, overlap trade-offs, and tuning against recall@k instead of blog defaults.
- [How to Choose an Embedding Model for RAG (and Version It Like a Schema)](https://changegamer.ai/articles/choosing-embedding-models.md): Embedding selection as an operations problem: the criteria that dominate total cost of ownership, the never-mix-spaces invariant, reindex migrations with dual indexes, and where quantization fits once cost shows up.
- [How to Combine Keyword and Vector Search in RAG](https://changegamer.ai/articles/hybrid-retrieval-fusion.md): Why hybrid retrieval is the production default rather than an upgrade: complementary failure modes of lexical and dense search, reciprocal rank fusion versus weighted scoring, parameter choices, and how filtering interacts with fusion.
- [How to Evaluate a RAG System (Retrieval Metrics, Generation Metrics, CI Gates)](https://changegamer.ai/articles/evaluating-rag-systems.md): The evaluation harness that keeps RAG changeable: golden-set construction, retrieval metrics separated from generation metrics, LLM-as-judge screening with human acceptance, CI regression gates, and the logged-query flywheel.
- [How to Keep a RAG Index Fresh (Staleness Bounds, Not Vibes)](https://changegamer.ai/articles/rag-index-freshness.md): Freshness as an engineered property: per-source staleness contracts, document versioning and tombstones, effective-date filtering, deletion propagation with reconciliation backstops, and the sync-lag metrics that predict stale answers before users report them.
- [Permission-Aware Retrieval: Multi-Tenant RAG Without Leaks](https://changegamer.ai/articles/multi-tenant-rag-permissions.md): How to enforce access control inside retrieval for multi-tenant RAG: authorization as mandatory pre-filters derived from the authenticated principal, tenant isolation mechanics, permission-negative testing, and the post-filter trap that produces empty answers.
- [Retrieval as a Tool: Agentic RAG Patterns That Survive Production](https://changegamer.ai/articles/agentic-retrieval-patterns.md): When AI agents consume retrieval as a tool rather than a pipeline stage: narrow tool contracts, hard budgets on iterative retrieval, query decomposition, provenance-carrying results, and keeping trust boundaries inside the retrieval path.
- [How to Cut RAG Cost and Latency Without Cutting Quality](https://changegamer.ai/articles/rag-cost-and-latency.md): RAG cost and latency as engineered budgets: where the money actually goes, caching layers and their hit-rate economics, routing queries to right-sized models, bounding retrieval fan-out, the hidden lines (reindex migrations, eval compute), and p95 discipline.
- [Common RAG Failure Modes and How to Fix Them](https://changegamer.ai/articles/rag-failure-modes-runbook.md): An operator runbook for the four RAG failure classes with no dedicated deep-dive elsewhere: retrieval miss, context overload, injection via content at ingestion time, and silent quality degradation — symptom, first diagnostic, and fix for each.
- [GraphRAG vs Vector RAG: When to Use a Knowledge Graph Instead](https://changegamer.ai/articles/graphrag-vs-vector-rag.md): A decision framework for choosing graph-structured retrieval over standard vector RAG: which query types GraphRAG actually wins, what building a knowledge graph costs, named implementations, and hybrid vector-plus-graph patterns.
- [When Is RAG the Wrong Answer? A Decision Guide](https://changegamer.ai/articles/when-rag-is-the-wrong-answer.md): A worked decision guide for the four real alternatives to retrieval-augmented generation: including knowledge directly, querying structured data with text-to-SQL, fine-tuning for behavior change, and graph-based retrieval for entity relationships.

## Reference resources

- https://changegamer.ai/resources/reranking-for-rag.md
- https://changegamer.ai/resources/hybrid-search-for-rag.md
- https://changegamer.ai/resources/agent-cost-latency-optimization.md
- https://changegamer.ai/resources/evaluating-ai-agents.md
- https://changegamer.ai/resources/rag-retrieval-for-agents.md

All guides: https://changegamer.ai/api/articles.json · Reference corpus: https://changegamer.ai/llms.txt
Licensing: https://changegamer.ai/api/pricing.json (offer catalog) · https://changegamer.ai/api/payment.json (payment methods, HTTP 402 flow) · access guide: https://changegamer.ai/resources/access-and-pricing.md