# Reranking for RAG: Cross-Encoders, LLM Rerankers, and Hosted APIs

> Second-stage retrieval step that re-scores bi-encoder candidates with full query-document attention, boosting precision without sacrificing recall; covers cross-encoder, LLM, and late-interaction reranking, hosted APIs, sizing heuristics, and evaluation.

Category: Guide · Updated: 2026-06-25 · Tags: rag, retrieval, reranking, cross-encoder, embeddings, vector-search, evaluation, agents
Canonical: https://changegamer.ai/resources/reranking-for-rag
Variants: [HTML](https://changegamer.ai/resources/reranking-for-rag) · [JSON](https://changegamer.ai/api/resources/reranking-for-rag.json)
License: https://changegamer.ai/license.xml · Access: free

Bi-encoder retrieval (ANN over embeddings) maximises recall cheaply but scores each document independently, missing subtle query-document interactions. A reranker runs a second pass over the top-N candidates and reorders them with full joint attention, recovering precision without rescanning the full index.

## Why rerank? Recall vs. precision

Bi-encoder ANN retrieval optimises recall@k: find everything plausibly relevant fast. Reranking optimises precision@k: put the most relevant chunks first so your LLM context is not diluted. The two-stage split lets each component do what it does best.

Standard sizing heuristic: retrieve top-50 to top-100 with ANN (cheap), rerank to top-5 to top-20 (expensive). Retrieving too few limits recall; reranking too many exceeds latency budgets. Tune against your query latency SLA and your recall@k evaluation scores.

## What architectures exist?

**Cross-encoder rerankers** — encode the query and each candidate *together* in a single forward pass (e.g., BERT-style model with a [CLS] → relevance score head). Full token-level cross-attention captures interactions that separate embeddings miss. Accuracy is high; inference is O(N × model_cost) over N candidates. Cross-encoders are the dominant deployed approach.

**LLM rerankers** — prompt a large language model to score or sort candidates (pointwise: "rate relevance 1-10"; listwise: "rank these passages"). Highest semantic understanding; also highest latency and cost. Practical mainly when candidate N is small (≤10) or quality trumps cost.

**Late-interaction (ColBERT-style)** — encode query and document independently to *per-token* vectors; score via MaxSim (maximum inner product across token pairs). Decompresses to near-cross-encoder quality while precomputing document token embeddings offline. The PLAID engine (arXiv:2205.09707) reduces late-interaction search latency substantially versus naïve ColBERTv2 via centroid pruning. Acts more as a first-stage dense retriever than a pure reranker, but fills the gap between bi-encoders and cross-encoders.

## Comparison table

| Approach | Relative latency | Relative quality | Hosted API? |
|---|---|---|---|
| Cross-encoder (open-weight, e.g. BGE-reranker-v2-m3) | Low (self-hosted GPU) | High | No (open-weight) |
| Cross-encoder (hosted, e.g. Cohere Rerank 4, Voyage rerank-2.5) | Low–medium | High | Yes |
| LLM reranker (listwise, frontier model) | High | Highest | Yes |
| Late-interaction / ColBERT (PLAID) | Low at scale (GPU) | High | Limited |

Latency is workload-dependent — benchmark on your own data and hardware before committing.

## Hosted rerank APIs (verified)

- **Cohere Rerank** — `rerank-v4.0-pro` and `rerank-v4.0-fast` (Rerank 4, with a 32K-token context, ~4× the v3.5 window). Single multilingual model; available direct and on Azure AI Foundry. Docs: https://docs.cohere.com/docs/rerank-overview
- **Voyage AI** — `rerank-2.5` and `rerank-2.5-lite` (instruction-following rerankers; multilingual; among the lowest-latency hosted rerankers). Docs: https://docs.voyageai.com/docs/reranker
- **Jina AI** — `jina-reranker-v3` via hosted API; earlier versions are also open-weight on Hugging Face. Docs: https://jina.ai/reranker/

## Open-weight rerankers (self-hosted)

- `BAAI/bge-reranker-v2-m3` — multilingual cross-encoder on the BGE M3 backbone; use via the FlagEmbedding library.
- `colbert-ir/colbertv2.0` — ColBERT late-interaction model; use with the PLAID engine for large-scale retrieval.

## How should agents call a reranker?

Expose retrieve-then-rerank as one tool so the agent does not orchestrate two separate calls: the tool takes a query and returns ranked chunks with scores. Set a score threshold or a hard top-k limit before injecting into context to prevent overflow. See /resources/rag-retrieval-for-agents for the full pipeline and /resources/embeddings-vector-search for the ANN retrieval stage that feeds the reranker.

## Evaluation

Measure reranker impact in isolation: fix the first-stage retrieval, vary the reranker, and compare NDCG@k (normalised discounted cumulative gain) or MRR (mean reciprocal rank) on a labelled query set. BEIR (arXiv:2104.08663) is the standard zero-shot retrieval benchmark; many reranker papers report BEIR scores for comparability. For end-to-end RAG evaluation, use RAGAS (github.com/explodinggradients/ragas, arXiv:2309.15217).

## Verified sources

- Cohere Rerank overview: https://docs.cohere.com/docs/rerank-overview
- Voyage AI reranker docs: https://docs.voyageai.com/docs/reranker
- Jina Reranker: https://jina.ai/reranker/
- BGE reranker-v2-m3 (Hugging Face): https://huggingface.co/BAAI/bge-reranker-v2-m3
- ColBERTv2 (Hugging Face): https://huggingface.co/colbert-ir/colbertv2.0
- ColBERT paper (arXiv:2004.12832, SIGIR 2020): https://arxiv.org/abs/2004.12832
- PLAID efficient late-interaction retrieval (arXiv:2205.09707): https://arxiv.org/abs/2205.09707
- BEIR retrieval benchmark (arXiv:2104.08663): https://arxiv.org/abs/2104.08663
- RAGAS paper (arXiv:2309.15217): https://arxiv.org/abs/2309.15217

---

## Related resources

- [RAG and Retrieval for Agents](https://changegamer.ai/resources/rag-retrieval-for-agents.md): End-to-end practitioner reference for Retrieval-Augmented Generation: pipeline stages, chunking strategies, dense/sparse/hybrid retrieval, reranking, agentic retrieval patterns, quality failure modes, and evaluation — with verified sources for every named technique.
- [Hybrid Search for RAG: BM25 + Dense Retrieval and Fusion](https://changegamer.ai/resources/hybrid-search-for-rag.md): How to combine lexical (BM25/SPLADE) and dense vector retrieval with Reciprocal Rank Fusion for higher first-stage recall in RAG pipelines — with the RRF formula, a sparse-method comparison table, and verified DB support.
- [Embeddings and Vector Search for Agents](https://changegamer.ai/resources/embeddings-vector-search.md): How to pick an embedding model, understand distance metrics, choose an ANN index type, and operate a vector store reliably in agent retrieval pipelines.
- [Chunking Strategies for RAG](https://changegamer.ai/resources/chunking-strategies-for-rag.md): Practitioner reference for chunking documents before embedding: fixed-size, recursive, semantic, late chunking, and contextual retrieval — with a strategy comparison table, chunk-size and overlap tradeoffs, code/table/Markdown handling, embedding model context limits, and evaluation methods.

---

Index of all resources: https://changegamer.ai/llms.txt · Full corpus: https://changegamer.ai/llms-full.txt · Corpus data (NDJSON): https://changegamer.ai/api/corpus.jsonl · Offers: https://changegamer.ai/api/pricing.json