← All guides · RAG in production

How to Combine Keyword and Vector Search in RAG

Part 4 of RAG in production · 914 words · ~4 min read · published 2026-08-23 · updated 2026-08-23 · Markdown variant

Why hybrid retrieval is the production default rather than an upgrade: complementary failure modes of lexical and dense search, reciprocal rank fusion versus weighted scoring, parameter choices, and how filtering interacts with fusion.

Part of the Agentic RAG in Production: The Complete Operator Guide guide.

The pillar guide declares hybrid retrieval the default; this article justifies that stance and turns it into parameters. The core argument is about failure modes, not averages: neither channel is merely "worse" than the other — each fails on different inputs, which is exactly what makes their combination powerful.

Why do keyword and vector search fail differently?

Lexical methods such as BM25 match terms with statistics — term frequency, document length, rarity. They excel where the query shares surface form with the answer: error codes, SKUs, API names, acronyms, statutes. They fail under paraphrase and vocabulary mismatch, penalizing well-written answers phrased differently from the question.

Dense retrieval embeds query and passages into one semantic space, matching meaning rather than strings. It catches paraphrase, synonyms and cross-lingual matches. It blurs exact tokens — two similar-looking identifiers embed almost identically — and it inherits whatever biases its training distribution carried.

The failure sets barely overlap. A help desk query citing an exact exception code defeats dense retrieval trivially and lexical trivially well; "how do I stop my worker dying after deploy" defeats lexical if the docs say "crash loop post-release" and falls easily to embeddings. Production traffic contains generous amounts of both kinds.

Which fusion method should you use?

Two workhorses dominate practice:

• Reciprocal Rank Fusion — merge by ranks alone: each document gains 1 / (k + rank) per list, k commonly set around 60. No score normalization needed, no tuning beyond k, remarkably hard to break. Start here.
• Weighted score fusion — normalize each channel's scores, then blend with weights. Finer control (down-weighting lexical for chatty paraphrase-heavy traffic, up-weighting it for code search) but the normalization is a maintenance liability: score distributions shift with corpus growth and model changes.

Whichever you pick, evaluate fusion as a measured decision on your golden set — the same harness used for chunking and embedding decisions — rather than a preference.

What parameters actually move the needle?

Three, in order of impact:

1. Candidates per channel: how deep each channel retrieves before fusion. Too shallow caps quality — the true best passage may never enter any list. Plot final answer metric against candidate depth; choose at the knee, not the asymptote.
2. Final selection size: how many fused results survive to reranking or assembly. This interacts with downstream budget (see the corpus reranking reference for the split).
3. Fusion constant or weights: real but smallest effect. Tune last.

How do you prove hybrid actually helped?

Run the counterfactual rather than the vibe: the golden-set harness scores three configurations — lexical only, dense only, fused — on identical questions. The interesting output is not the average uplift but the per-question diff: which identifier-style questions flipped from miss to hit, and whether any paraphrase questions regressed. Keep that diff list; it becomes regression coverage for the fusion parameters themselves. In production, log which channel contributed each selected passage (the fusion step knows each item's origin) so per-channel contribution stays observable as your query mix drifts.

What breaks hybrid systems in production?

Four recurring failure modes deserve standing checks:

1. Channel skew — one channel's index lags the other after partial ingestion failures, so fusion blends evidence from different corpus generations. Monitor document counts and newest synced versions across both indexes together.
2. Score-normalization drift — weighted-fusion setups calibrated months ago silently misbehave as corpus growth shifts score distributions. RRF avoids this by construction; weighted setups need periodic recalibration against the golden set.
3. Silent channel degradation — a broken lexical analyzer or an embedding API change can halve one channel while the system still looks alive overall. Track per-channel hit rates; a sudden monopoly in contribution is an alarm, not a win.
4. Filter asymmetry — permissions enforced on one channel but not the other turn hybrid into a leak vector. The pre-filter rule must hold identically everywhere, verified by permission-negative test cases that run both channels.

How does this compose with filters, permissions and reranking?

Order of operations is the whole game:

• Permissions and scope filters apply pre-fusion, inside each channel, so every candidate is already legitimate
• Fusion merges the filtered lists
• Deduplication collapses near-identical passages that arrived via both channels
• Reranking then reorders survivors if budgeted

Skip the pre-fusion filtering and you invite the empty-list failure: all top fused hits from one channel, all filtered out afterward. Skip deduplication and the generator sees the same passage twice with different scores, overweighting it for no informational gain. The vector database reference matters here too — stores differ sharply in how efficiently they serve filtered vector search, and that difference lands directly on your most permission-sensitive queries.

Adopt hybrid early, measure the parameters honestly, and the system quietly stops failing on exact-string questions while keeping everything embeddings are good at.

If you take one number away, take this: log per-question channel attribution from day one. Every later decision in this article's lifecycle — fusion weights, candidate depth, whether to gate the second channel, when to suspect skew — is answered in minutes with that log and argued about for weeks without it. Hybrid retrieval is cheap to build and easy to misjudge; attribution data is what keeps the judgment honest. It also settles the most common post-launch argument before it starts: when someone proposes removing a channel for simplicity, the contribution log shows exactly which classes of questions would regress, and the decision becomes arithmetic instead of taste.

#rag #retrieval #search #bm25 #embeddings