Building Production-Ready RAG: Hybrid Search, Reranking, and Conversational Memory

Building Production-Ready RAG: Hybrid Search, Reranking, and Conversational Memory

The gap between a working RAG demo and a production system is not model quality; it is three specific capabilities that demos skip. Demo pipelines use pure vector search, which misses exact terms like error codes and SKU strings. They trust top-k blindly, which surfaces right-topic-wrong-answer chunks. And they treat every question as the first sentence ever spoken, which collapses the moment a user says "what about the second option?"

This article covers the upgrades that close those gaps: hybrid retrieval fusing dense embeddings with lexical BM25, a reranking stage that reorders candidates with a better-informed model, and conversational memory patterns that keep multi-turn dialogue coherent without drowning your context window. Each one earns its complexity with measurable retrieval improvements, and each has a failure mode worth knowing before you ship it.

What "Production" Adds#

A production RAG system answers four questions the demo never asked. Did we find documents containing the exact identifiers users typed? Are the chunks we retrieved actually answer-capable, or merely nearby? Does the third follow-up question still resolve to what the user meant? And can we observe any of this when quality silently degrades at 2am? The sections below are the standard answers, ordered by return on investment based on what actually moves hit rates in shipped systems.

Hybrid Search: Dense Meets Lexical#

Pure embedding search has a documented blind spot: it matches meaning but mangles exactness. Ask for "error E-4021" and the nearest neighbors are other error-handling discussions, not the page defining that specific code. Lexical search (BM25, the algorithm behind classic search engines) does the opposite: precise on rare tokens and identifiers, blind to paraphrase.

Hybrid retrieval runs both and merges:

from rank_bm25 import BM25Okapi
import numpy as np

class HybridRetriever:
    def __init__(self, chunks: list[dict], vectors: np.ndarray):
        self.chunks = chunks
        self.vectors = vectors                      # pre-normalized embeddings
        tokenized = [c["text"].lower().split() for c in chunks]
        self.bm25 = BM25Okapi(tokenized)

    def dense_scores(self, query_vec) -> np.ndarray:
        return self.vectors @ query_vec             # cosine via dot product

    def lexical_scores(self, query: str) -> np.ndarray:
        return np.array(self.bm25.get_scores(query.lower().split()))

    def search(self, query: str, query_vec, k: int = 10):
        dense = _rank_positions(self.dense_scores(query_vec))
        lex = _rank_positions(self.lexical_scores(query))
        fused = _rrf(dense, lex, k=k)
        return [self.chunks[i] | {"score": s} for i, s in fused]

The two ranking lists need fusion, and that is where teams historically lost weeks tuning weighted blends like 0.7 * dense + 0.3 * bm25, which requires normalizing incomparable score scales. Reciprocal Rank Fusion solves it with no tuning at all: score each document by sum over lists of 1/(k + rank) using the standard constant k=60. Rank position is scale-free by construction:

def _rank_positions(scores: np.ndarray) -> list[int]:
    return list(np.argsort(scores)[::-1])

def _rrf(*rank_lists, k: int = 60, top_n: int = 50):
    fused: dict[int, float] = {}
    for ranks in rank_lists:
        for rank_pos, doc_idx in enumerate(ranks[:top_n]):
            fused[doc_idx] = fused.get(doc_idx, 0.0) + 1.0 / (k + rank_pos + 1)
    return sorted(fused.items(), key=lambda kv: kv[1], reverse=True)

The diagram understates how well this composes: because fusion operates on ranks, you can add more retrievers later (a filtered-by-permission pass, a same-author boost) without re-tuning weights.

Watching the fusion happen on concrete ranks makes the math obvious:

Doc A appears high on both lists and wins; Doc F, a dense-only favorite with no lexical support, sinks below documents that both retrievers respect. That consensus behavior is exactly what you want from identifiers-and-paraphrase queries alike.

Reranking: The Cheap Quality Multiplier#

Here is an uncomfortable truth about first-stage retrieval: bi-encoder embeddings compress an entire chunk into one vector, so they score relevance with limited resolution. Cross-encoder rerankers fix this by reading query and candidate together, jointly attending across both texts, and scoring true relevance with far higher fidelity. The catch is cost: cross-encoding is quadratic-ish in attention over both sequences, so you cannot run it against a million documents. You run it against fifty.

That asymmetry defines the architecture: retrieve cheaply and broadly (top 25-100 candidates via hybrid search), then rerank down to the five that enter the prompt:

import cohere

co = cohere.Client("your-api-key")

def rerank(query: str, candidates: list[dict], top_n: int = 5) -> list[dict]:
    resp = co.rerank(
        model="rerank-v3.5",
        query=query,
        documents=[c["text"] for c in candidates],
        top_n=top_n,
    )
    out = []
    for r in resp.results:
        cand = candidates[r.index]
        out.append(cand | {"relevance": r.relevance_score})
    return out

Self-hosted alternative if data cannot leave your perimeter or call volume makes per-query pricing painful: sentence_transformers.CrossEncoder with an MS MARCO-trained checkpoint like cross-encoder/ms-marco-MiniLM-L-6-v2, scoring (query, passage) pairs locally on a modest GPU.

Reranking reliably delivers the largest single-jump retrieval improvement of anything in this article precisely because it corrects ordering errors rather than recall errors. It cannot rescue a candidate that hybrid search never surfaced, which is why stage one must stay broad.

Conversational Memory Without Context Bloat#

Multi-turn RAG breaks at turn two. User asks "what's the refund policy?", gets an answer, then asks "what about international orders?" Embedded alone, that follow-up retrieves garbage about geography because its antecedent lives in the previous turn. Stuffing full history into the retrieval query pollutes the embedding; stuffing it into the prompt bloats context and bills every turn.

Two mechanisms solve this cleanly, and they compose.

Condense-then-retrieve: rewrite the follow-up into a standalone question before retrieval, using history as context for the rewriting only:

CONDENSE_PROMPT = """Given the conversation and a follow-up question,
rewrite the follow-up as a standalone question that includes any
needed references. Reply with only the rewritten question.

Chat history:
{history}

Follow-up: {question}"""

def standalone_question(history: str, question: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0.0,
        messages=[{"role": "user",
                   "content": CONDENSE_PROMPT.format(
                       history=history, question=question)}],
    )
    return resp.choices[0].message.content or question

Now retrieval sees "What is the refund policy for international orders?" and finds the right chunks. The rewrite costs one small-model call per turn, trivially cached.

Keep the rewrite conservative: the rewriter's job is resolving references, not expanding scope. A rewrite that invents constraints the user never stated will retrieve confidently for a question nobody asked, which is worse than a vague query retrieving vaguely.

Tiered history management: keep the recent few turns verbatim (tone and pronouns live there), compress older turns into a running summary, and store durable facts (names, order numbers, stated preferences) as structured slots rather than prose:

class ConversationMemory:
    def __init__(self, window: int = 4):
        self.turns: list[tuple[str, str]] = []     # (question, answer)
        self.summary: str = ""
        self.facts: dict[str, str] = {}            # extracted slots
        self.window = window

    def context_for_rewrite(self) -> str:
        recent = [f"Q: {q}\nA: {a}" for q, a in self.turns[-self.window:]]
        parts = ([f"Summary of earlier conversation:\n{self.summary}"]
                 if self.summary else [])
        parts += recent + [f"Known facts: {self.facts}" if self.facts else ""]
        return "\n".join(p for p in parts if p)

    def append(self, q: str, a: str):
        self.turns.append((q, a))
        if len(self.turns) > 12:                    # fold into summary
            folded = self.turns[:-self.window]
            self.summary = summarize_turns(self.summary, folded)
            self.turns = self.turns[-self.window:]

The summarization call runs rarely and off the critical path; the facts dictionary is where extraction endpoints shine, since order numbers and account types belong in typed slots, not buried in paragraphs a rewrite step might garble.

The Operational Layer Nobody Writes Blog Posts About#

Three unglamorous capabilities decide whether any of the above survives contact with reality. Caching: repeated questions deserve identical cached answers keyed on the rewritten query, cutting latency to milliseconds and cost to zero for your most common traffic. The key must include permission scope, or one user's authorized answer becomes another user's data leak:

import hashlib
import json

def cache_key(rewritten_query: str, permission_scope: str) -> str:
    payload = json.dumps([rewritten_query, permission_scope],
                         sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

Freshness: documents change; rebuild affected index entries on change events, even crudely, because a confidently cited deleted policy is worse than no answer. Access control belongs inside retrieval, not around it: filter at query time by permission metadata so a sales rep cannot retrieve HR-only chunks through clever phrasing, since post-filtering the generated answer is already too late.

Observability closes the loop. Log per query: rewritten question, retrieved sources with scores, reranked order, token counts, and whether generation refused. Weekly, sample real traffic and grade answers against sources; quality regressions announce themselves in that log long before customers file tickets. Two derived metrics deserve dashboards from day one: refusal rate, which spikes when either the index went stale or a threshold got tightened, and p95 end-to-end latency broken down by stage (rewrite, retrieve, rerank, generate), because "RAG is slow" is never actionable until you know which box ate the budget.

Failure Modes You Will Meet in Month One#

Each upgrade carries its own traps. Hybrid search surfaces keyword-heavy spam chunks that BM25 loves but humans ignore; the reranker usually absorbs these, but watch for them when reranking is disabled behind a feature flag. RRF fusion can promote mediocre middle-of-list results over strong specialists; if one corpus section consistently wins wrongly, check for duplicated near-identical chunks inflating its rank counts. Reranking adds 100-300ms per query with hosted APIs; budget p99 accordingly or cache aggressively, because users feel reranking even when they cannot name it. Condense-rewrite occasionally hallucinates details into the standalone question that were never asked, especially with aggressive summaries; keep rewrites conservative and log originals alongside rewrites so drift is auditable. Rolling summaries lose detail by design, and support conversations where the user mentioned an order number eight turns ago will surface exactly when your summary dropped it; that is what the structured facts tier exists to prevent, so feed it deliberately.

Two more show up reliably once real users arrive. First, permission drift: someone reorganizes document storage, metadata stops matching the access-control filter values, and retrieval silently returns nothing for whole teams while generation politely refuses; treat filter keys as schema and test them alongside code. Second, threshold rot: score distributions shift when you swap embedding models or chunk sizes, and a refusal cutoff tuned for the old vectors starts rejecting everything; recalibrate thresholds on your golden set as part of any component change rather than discovering them through refusal-rate alarms at midnight. Both failures are silent by nature, which is precisely why the observability metrics above are not optional infrastructure.

Boring Systems Win. Production RAG is not exotic: lexical plus vector retrieval fused by rank math, a reranker standing between candidates and the prompt, memory split across verbatim windows, summaries, and typed slots, all wrapped in caching, permissions, and logs. Every piece here predates large language models; the discipline is wiring them where their failure modes do not compound. Teams that resist adding machinery beyond what their hit-rate measurements demand end up with systems that quietly keep working, which is the entire definition of production.