{
  "slug": "reranking-for-rag",
  "title": "Reranking for RAG: Cross-Encoders, LLM Rerankers, and Hosted APIs",
  "description": "Second-stage retrieval step that re-scores bi-encoder candidates with full query-document attention, boosting precision without sacrificing recall; covers cross-encoder, LLM, and late-interaction reranking, hosted APIs, sizing heuristics, and evaluation.",
  "category": "Guide",
  "tags": [
    "rag",
    "retrieval",
    "reranking",
    "cross-encoder",
    "embeddings",
    "vector-search",
    "evaluation",
    "agents"
  ],
  "updated": "2026-06-25",
  "canonical": "https://changegamer.ai/resources/reranking-for-rag",
  "markdown": "https://changegamer.ai/resources/reranking-for-rag.md",
  "outline": [
    {
      "depth": 2,
      "text": "Why rerank? Recall vs. precision",
      "anchor": "why-rerank-recall-vs-precision"
    },
    {
      "depth": 2,
      "text": "What architectures exist?",
      "anchor": "what-architectures-exist"
    },
    {
      "depth": 2,
      "text": "Comparison table",
      "anchor": "comparison-table"
    },
    {
      "depth": 2,
      "text": "Hosted rerank APIs (verified)",
      "anchor": "hosted-rerank-apis-verified"
    },
    {
      "depth": 2,
      "text": "Open-weight rerankers (self-hosted)",
      "anchor": "open-weight-rerankers-self-hosted"
    },
    {
      "depth": 2,
      "text": "How should agents call a reranker?",
      "anchor": "how-should-agents-call-a-reranker"
    },
    {
      "depth": 2,
      "text": "Evaluation",
      "anchor": "evaluation"
    },
    {
      "depth": 2,
      "text": "Verified sources",
      "anchor": "verified-sources"
    }
  ],
  "related": [
    {
      "slug": "rag-retrieval-for-agents",
      "title": "RAG and Retrieval for Agents",
      "description": "End-to-end practitioner reference for Retrieval-Augmented Generation: pipeline stages, chunking strategies, dense/sparse/hybrid retrieval, reranking, agentic retrieval patterns, quality failure modes, and evaluation — with verified sources for every named technique.",
      "url": "https://changegamer.ai/resources/rag-retrieval-for-agents"
    },
    {
      "slug": "hybrid-search-for-rag",
      "title": "Hybrid Search for RAG: BM25 + Dense Retrieval and Fusion",
      "description": "How to combine lexical (BM25/SPLADE) and dense vector retrieval with Reciprocal Rank Fusion for higher first-stage recall in RAG pipelines — with the RRF formula, a sparse-method comparison table, and verified DB support.",
      "url": "https://changegamer.ai/resources/hybrid-search-for-rag"
    },
    {
      "slug": "embeddings-vector-search",
      "title": "Embeddings and Vector Search for Agents",
      "description": "How to pick an embedding model, understand distance metrics, choose an ANN index type, and operate a vector store reliably in agent retrieval pipelines.",
      "url": "https://changegamer.ai/resources/embeddings-vector-search"
    },
    {
      "slug": "chunking-strategies-for-rag",
      "title": "Chunking Strategies for RAG",
      "description": "Practitioner reference for chunking documents before embedding: fixed-size, recursive, semantic, late chunking, and contextual retrieval — with a strategy comparison table, chunk-size and overlap tradeoffs, code/table/Markdown handling, embedding model context limits, and evaluation methods.",
      "url": "https://changegamer.ai/resources/chunking-strategies-for-rag"
    }
  ],
  "body": "Bi-encoder retrieval (ANN over embeddings) maximises recall cheaply but scores each document independently, missing subtle query-document interactions. A reranker runs a second pass over the top-N candidates and reorders them with full joint attention, recovering precision without rescanning the full index.\n\n## Why rerank? Recall vs. precision\n\nBi-encoder ANN retrieval optimises recall@k: find everything plausibly relevant fast. Reranking optimises precision@k: put the most relevant chunks first so your LLM context is not diluted. The two-stage split lets each component do what it does best.\n\nStandard sizing heuristic: retrieve top-50 to top-100 with ANN (cheap), rerank to top-5 to top-20 (expensive). Retrieving too few limits recall; reranking too many exceeds latency budgets. Tune against your query latency SLA and your recall@k evaluation scores.\n\n## What architectures exist?\n\n**Cross-encoder rerankers** — encode the query and each candidate *together* in a single forward pass (e.g., BERT-style model with a [CLS] → relevance score head). Full token-level cross-attention captures interactions that separate embeddings miss. Accuracy is high; inference is O(N × model_cost) over N candidates. Cross-encoders are the dominant deployed approach.\n\n**LLM rerankers** — prompt a large language model to score or sort candidates (pointwise: \"rate relevance 1-10\"; listwise: \"rank these passages\"). Highest semantic understanding; also highest latency and cost. Practical mainly when candidate N is small (≤10) or quality trumps cost.\n\n**Late-interaction (ColBERT-style)** — encode query and document independently to *per-token* vectors; score via MaxSim (maximum inner product across token pairs). Decompresses to near-cross-encoder quality while precomputing document token embeddings offline. The PLAID engine (arXiv:2205.09707) reduces late-interaction search latency substantially versus naïve ColBERTv2 via centroid pruning. Acts more as a first-stage dense retriever than a pure reranker, but fills the gap between bi-encoders and cross-encoders.\n\n## Comparison table\n\n| Approach | Relative latency | Relative quality | Hosted API? |\n|---|---|---|---|\n| Cross-encoder (open-weight, e.g. BGE-reranker-v2-m3) | Low (self-hosted GPU) | High | No (open-weight) |\n| Cross-encoder (hosted, e.g. Cohere Rerank 4, Voyage rerank-2.5) | Low–medium | High | Yes |\n| LLM reranker (listwise, frontier model) | High | Highest | Yes |\n| Late-interaction / ColBERT (PLAID) | Low at scale (GPU) | High | Limited |\n\nLatency is workload-dependent — benchmark on your own data and hardware before committing.\n\n## Hosted rerank APIs (verified)\n\n- **Cohere Rerank** — `rerank-v4.0-pro` and `rerank-v4.0-fast` (Rerank 4, with a 32K-token context, ~4× the v3.5 window). Single multilingual model; available direct and on Azure AI Foundry. Docs: https://docs.cohere.com/docs/rerank-overview\n- **Voyage AI** — `rerank-2.5` and `rerank-2.5-lite` (instruction-following rerankers; multilingual; among the lowest-latency hosted rerankers). Docs: https://docs.voyageai.com/docs/reranker\n- **Jina AI** — `jina-reranker-v3` via hosted API; earlier versions are also open-weight on Hugging Face. Docs: https://jina.ai/reranker/\n\n## Open-weight rerankers (self-hosted)\n\n- `BAAI/bge-reranker-v2-m3` — multilingual cross-encoder on the BGE M3 backbone; use via the FlagEmbedding library.\n- `colbert-ir/colbertv2.0` — ColBERT late-interaction model; use with the PLAID engine for large-scale retrieval.\n\n## How should agents call a reranker?\n\nExpose retrieve-then-rerank as one tool so the agent does not orchestrate two separate calls: the tool takes a query and returns ranked chunks with scores. Set a score threshold or a hard top-k limit before injecting into context to prevent overflow. See /resources/rag-retrieval-for-agents for the full pipeline and /resources/embeddings-vector-search for the ANN retrieval stage that feeds the reranker.\n\n## Evaluation\n\nMeasure reranker impact in isolation: fix the first-stage retrieval, vary the reranker, and compare NDCG@k (normalised discounted cumulative gain) or MRR (mean reciprocal rank) on a labelled query set. BEIR (arXiv:2104.08663) is the standard zero-shot retrieval benchmark; many reranker papers report BEIR scores for comparability. For end-to-end RAG evaluation, use RAGAS (github.com/explodinggradients/ragas, arXiv:2309.15217).\n\n## Verified sources\n\n- Cohere Rerank overview: https://docs.cohere.com/docs/rerank-overview\n- Voyage AI reranker docs: https://docs.voyageai.com/docs/reranker\n- Jina Reranker: https://jina.ai/reranker/\n- BGE reranker-v2-m3 (Hugging Face): https://huggingface.co/BAAI/bge-reranker-v2-m3\n- ColBERTv2 (Hugging Face): https://huggingface.co/colbert-ir/colbertv2.0\n- ColBERT paper (arXiv:2004.12832, SIGIR 2020): https://arxiv.org/abs/2004.12832\n- PLAID efficient late-interaction retrieval (arXiv:2205.09707): https://arxiv.org/abs/2205.09707\n- BEIR retrieval benchmark (arXiv:2104.08663): https://arxiv.org/abs/2104.08663\n- RAGAS paper (arXiv:2309.15217): https://arxiv.org/abs/2309.15217",
  "sources": [
    "https://docs.cohere.com/docs/rerank-overview",
    "https://docs.voyageai.com/docs/reranker",
    "https://jina.ai/reranker/",
    "https://huggingface.co/BAAI/bge-reranker-v2-m3",
    "https://huggingface.co/colbert-ir/colbertv2.0",
    "https://arxiv.org/abs/2004.12832",
    "https://arxiv.org/abs/2205.09707",
    "https://arxiv.org/abs/2104.08663",
    "https://arxiv.org/abs/2309.15217"
  ]
}