Advanced RAG — Reranking, Hybrid Search and Evaluation

A good research librarian does two things a bad one does not. First: when you ask "find me information about ACH payment limits," they search both the subject index (semantic) and the keyword catalogue (exact match) — not just one. Second: after gathering candidates, they skim each one to pick the three most directly relevant — they rerank. A naive RAG pipeline skips both steps. Hybrid search is the librarian searching two catalogues. Reranking is the librarian reading before recommending.

Adding a reranker alone typically improves end-to-end RAG quality by 10–25% with minimal engineering effort. It is the single highest-leverage improvement you can make to a naive RAG system.

import numpy as np
import faiss
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
import re

# ── Stripe knowledge base ───────────────────────────────────────────
KB_CHUNKS = [
    {'id': '1', 'text': 'Stripe domestic payments settle within T+2 business days. Settlement excludes weekends and public holidays.'},
    {'id': '2', 'text': 'International payments on Stripe settle within T+7 business days with SWIFT charges of USD 15-25.'},
    {'id': '3', 'text': 'Error code BAD_REQUEST_ERROR occurs when mandatory payment parameters are missing or invalid in the API call.'},
    {'id': '4', 'text': 'Error code GATEWAY_ERROR indicates the bank gateway timed out. Retry the payment after 5 minutes.'},
    {'id': '5', 'text': 'Error code BAD_REQUEST_ERROR with description invalid_api_key means the API key is incorrect or expired.'},
    {'id': '6', 'text': 'Refunds for ACH payments take 2-3 business days. Credit card refunds take 5-7 business days.'},
    {'id': '7', 'text': 'Webhook signature verification uses HMAC SHA256. Compute the signature using the webhook secret from dashboard.'},
    {'id': '8', 'text': 'The Stripe payment link expires after 15 minutes by default. Custom expiry can be set via the API.'},
    {'id': '9', 'text': 'Stripe supports ACH transfers, credit cards, debit cards, and digital wallets as payment methods.'},
    {'id': '10', 'text': 'To enable international payments on Stripe, submit KYC documents and business registration proof.'},
]

texts = [c['text'] for c in KB_CHUNKS]

# ── Dense retrieval — FAISS ───────────────────────────────────────────
embedder   = SentenceTransformer('all-MiniLM-L6-v2')
embeddings = embedder.encode(texts, normalize_embeddings=True)
dense_index = faiss.IndexFlatIP(embeddings.shape[1])
dense_index.add(embeddings)

def dense_search(query: str, k: int = 5) -> list[dict]:
    q_emb = embedder.encode([query], normalize_embeddings=True)
    scores, indices = dense_index.search(q_emb, k)
    return [{'id': KB_CHUNKS[i]['id'], 'rank': r+1, 'score': float(s)}
            for r, (i, s) in enumerate(zip(indices[0], scores[0]))]

# ── Sparse retrieval — BM25 ───────────────────────────────────────────
tokenised = [re.sub(r'[^ws]', '', t.lower()).split() for t in texts]
bm25      = BM25Okapi(tokenised)

def sparse_search(query: str, k: int = 5) -> list[dict]:
    q_tokens = re.sub(r'[^ws]', '', query.lower()).split()
    scores   = bm25.get_scores(q_tokens)
    top_idx  = scores.argsort()[::-1][:k]
    return [{'id': KB_CHUNKS[i]['id'], 'rank': r+1, 'score': float(scores[i])}
            for r, i in enumerate(top_idx)]

# ── Reciprocal Rank Fusion ────────────────────────────────────────────
def rrf_fusion(ranked_lists: list[list[dict]], k: int = 60) -> list[dict]:
    """
    Combine multiple ranked lists using Reciprocal Rank Fusion.
    ranked_lists: list of lists, each containing {'id': str, 'rank': int}
    Returns: merged list sorted by RRF score descending
    """
    rrf_scores: dict[str, float] = {}
    for ranked_list in ranked_lists:
        for item in ranked_list:
            doc_id = item['id']
            rank   = item['rank']
            rrf_scores[doc_id] = rrf_scores.get(doc_id, 0) + 1 / (k + rank)

    sorted_docs = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
    return [{'id': doc_id, 'rrf_score': score, 'rank': i+1}
            for i, (doc_id, score) in enumerate(sorted_docs)]

def hybrid_search(query: str, k_retrieve: int = 5, k_return: int = 3) -> list[dict]:
    """Full hybrid search pipeline."""
    dense_results  = dense_search(query, k=k_retrieve)
    sparse_results = sparse_search(query, k=k_retrieve)
    fused          = rrf_fusion([dense_results, sparse_results])

    # Attach chunk text to results
    id_to_chunk = {c['id']: c for c in KB_CHUNKS}
    return [
        {**r, 'text': id_to_chunk[r['id']]['text']}
        for r in fused[:k_return]
        if r['id'] in id_to_chunk
    ]

# ── Compare retrieval approaches ──────────────────────────────────────
test_queries = [
    ("What is error code BAD_REQUEST_ERROR?",
     "3",   # expects chunk about BAD_REQUEST_ERROR specifically
     "keyword-heavy — BM25 advantage"),
    ("How long until my money arrives after payment?",
     "1",   # expects settlement timeline chunk
     "semantic — dense advantage"),
    ("international payment setup",
     "10",  # expects KYC/international setup chunk
     "mixed"),
]

print(f"{'Query':<45} {'Dense':>8} {'Sparse':>8} {'Hybrid':>8} {'Expected'}")
print("─" * 75)
for query, expected_id, query_type in test_queries:
    dense_top  = dense_search(query, k=3)[0]['id']
    sparse_top = sparse_search(query, k=3)[0]['id']
    hybrid_top = hybrid_search(query, k_retrieve=5, k_return=3)[0]['id']

    d_mark = '✓' if dense_top  == expected_id else '✗'
    s_mark = '✓' if sparse_top == expected_id else '✗'
    h_mark = '✓' if hybrid_top == expected_id else '✗'

    print(f"  {query[:43]:<45} {d_mark}  #{dense_top}  {s_mark}  #{sparse_top}  {h_mark}  #{hybrid_top}  (exp #{expected_id})")

from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

# ── Load a cross-encoder reranker ────────────────────────────────────
# ms-marco models: trained on 500k (query, passage) pairs from MS MARCO dataset
# ms-marco-MiniLM-L-6-v2: fast, small (22M params)
# ms-marco-electra-base:   slower, more accurate (110M params)
reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def rerank(query: str, candidates: list[dict],
            top_k: int = 3) -> list[dict]:
    """
    Rerank retrieved candidates using a cross-encoder.
    candidates: list of {'id': str, 'text': str, ...}
    Returns top_k candidates sorted by cross-encoder relevance score.
    """
    if not candidates:
        return []

    # Cross-encoder scores each (query, passage) pair jointly
    pairs  = [(query, c['text']) for c in candidates]
    scores = reranker.predict(pairs)   # (N,) relevance scores

    # Sort by score descending
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [
        {**cand, 'rerank_score': float(score)}
        for cand, score in ranked[:top_k]
    ]

# ── Full two-stage pipeline ───────────────────────────────────────────
def two_stage_retrieve(query: str, top_k_coarse: int = 20,
                        top_k_final: int = 3) -> list[dict]:
    """
    Stage 1: hybrid search for top_k_coarse candidates
    Stage 2: cross-encoder reranking to top_k_final
    """
    # Stage 1: fast hybrid retrieval
    coarse = hybrid_search(query, k_retrieve=top_k_coarse,
                            k_return=top_k_coarse)
    # Stage 2: precise cross-encoder reranking
    reranked = rerank(query, coarse, top_k=top_k_final)
    return reranked

# ── Compare: naive vs two-stage ───────────────────────────────────────
queries = [
    "What does error code BAD_REQUEST_ERROR with invalid_api_key mean?",
    "When will my international payment settle?",
    "How do I verify webhook signatures?",
]

print("Two-stage retrieval results:")
for query in queries:
    print(f"
Query: '{query}'")

    # Naive: just dense search top-3
    naive    = dense_search(query, k=3)
    naive_ids = [r['id'] for r in naive]

    # Two-stage
    advanced = two_stage_retrieve(query, top_k_coarse=8, top_k_final=3)
    adv_ids  = [r['id'] for r in advanced]

    print(f"  Naive top-3:     chunks {naive_ids}")
    print(f"  Two-stage top-3: chunks {adv_ids}")

    # Show top reranked result
    if advanced:
        top = advanced[0]
        print(f"  Top result (score={top['rerank_score']:.3f}):")
        print(f"    '{top['text'][:80]}...'")

import torch
from sentence_transformers import SentenceTransformer
import faiss, numpy as np

embedder = SentenceTransformer('all-MiniLM-L6-v2')

# ── Technique 1: HyDE — Hypothetical Document Embeddings ─────────────
# Problem: short queries embed differently from long document chunks
# Fix: generate a hypothetical answer, embed that instead

def hyde_embed(query: str, llm_fn) -> np.ndarray:
    """
    Generate a hypothetical answer to the query,
    then embed the hypothetical answer instead of the query.
    The embedding will be stylistically similar to real document chunks.
    """
    hyde_prompt = (
        f"Write a 2-3 sentence passage that would directly answer "
        f"this question if it appeared in a technical documentation page:

"
        f"Question: {query}

Passage:"
    )
    # In production: call your LLM
    # hypothetical = llm_fn(hyde_prompt)
    # For demo: simulate what the hypothetical might look like
    hypothetical = f"[Hypothetical answer about: {query}. Would contain specific facts and terminology matching the query.]"

    emb = embedder.encode([hypothetical], normalize_embeddings=True)
    return emb

print("HyDE: embedding a hypothetical answer improves recall for short queries")
print("  Short query:     'settlement time'  → vague embedding")
print("  HyDE query:      'Payments settle within T+2 business days...' → precise")
print()

# ── Technique 2: Parent-child chunking ────────────────────────────────
# Problem: small chunks are precise but lack context for LLM
# Fix: index small child chunks, retrieve larger parent chunks

PARENT_CHUNKS = [
    {
        'id': 'P1',
        'text': (
            "Stripe Settlement Overview. "
            "Domestic payments settle within T+2 business days. "
            "International payments settle within T+7 business days. "
            "Settlement is initiated at 5 PM ET each business day. "
            "Minimum settlement amount is $100. "
            "Smaller amounts are carried forward to the next cycle."
        ),
        'children': [
            {'id': 'C1a', 'text': 'Domestic payments settle within T+2 business days.'},
            {'id': 'C1b', 'text': 'International payments settle within T+7 business days.'},
            {'id': 'C1c', 'text': 'Settlement is initiated at 5 PM ET each business day.'},
            {'id': 'C1d', 'text': 'Minimum settlement amount is $100.'},
        ],
    },
    {
        'id': 'P2',
        'text': (
            "Stripe Error Codes Reference. "
            "BAD_REQUEST_ERROR: mandatory parameters missing or invalid. "
            "GATEWAY_ERROR: bank gateway timeout, retry after 5 minutes. "
            "SERVER_ERROR: internal error, contact support if persistent. "
            "INVALID_SIGNATURE: webhook signature mismatch, check secret key."
        ),
        'children': [
            {'id': 'C2a', 'text': 'BAD_REQUEST_ERROR: mandatory parameters missing or invalid.'},
            {'id': 'C2b', 'text': 'GATEWAY_ERROR: bank gateway timeout, retry after 5 minutes.'},
            {'id': 'C2c', 'text': 'SERVER_ERROR: internal error, contact support if persistent.'},
            {'id': 'C2d', 'text': 'INVALID_SIGNATURE: webhook signature mismatch, check secret key.'},
        ],
    },
]

# Build index on child chunks
child_chunks = [c for p in PARENT_CHUNKS for c in p['children']]
child_to_parent = {c['id']: p['id'] for p in PARENT_CHUNKS for c in p['children']}
parent_map = {p['id']: p for p in PARENT_CHUNKS}

child_texts = [c['text'] for c in child_chunks]
child_embs  = embedder.encode(child_texts, normalize_embeddings=True)
child_index = faiss.IndexFlatIP(child_embs.shape[1])
child_index.add(child_embs)

def parent_child_search(query: str, k: int = 2) -> list[dict]:
    """Retrieve small child chunks, return their full parent chunks."""
    q_emb  = embedder.encode([query], normalize_embeddings=True)
    scores, idxs = child_index.search(q_emb, k * 2)   # over-retrieve children

    seen_parents = set()
    results = []
    for i, score in zip(idxs[0], scores[0]):
        child      = child_chunks[i]
        parent_id  = child_to_parent[child['id']]
        if parent_id not in seen_parents:
            seen_parents.add(parent_id)
            results.append({
                'parent_id':   parent_id,
                'matched_child': child['text'],
                'full_context':  parent_map[parent_id]['text'],
                'score':         float(score),
            })
        if len(results) >= k:
            break
    return results

query = "What does GATEWAY_ERROR mean?"
results = parent_child_search(query, k=2)
print(f"Parent-child search for: '{query}'")
for r in results:
    print(f"  Matched child:  '{r['matched_child']}'")
    print(f"  Full context returned to LLM:")
    print(f"    '{r['full_context'][:100]}...'")
    print()

# ── Technique 3: Query decomposition ─────────────────────────────────
print("Query decomposition:")
complex_query = "How do I set up Stripe for international payments and what fees will I pay?"
print(f"Complex query: '{complex_query}'")
print("Decompose into sub-queries:")
sub_queries = [
    "How do I enable international payments on Stripe?",
    "What are the fees for international payments on Stripe?",
]
for q in sub_queries:
    print(f"  → '{q}'")

import numpy as np
from sentence_transformers import SentenceTransformer, CrossEncoder
import re

embedder = SentenceTransformer('all-MiniLM-L6-v2')

# ── Faithfulness — is the answer grounded in context? ─────────────────
def faithfulness_score(answer: str, contexts: list[str],
                        nli_model=None) -> float:
    """
    Measure what fraction of answer sentences are supported by context.
    Uses cross-encoder NLI model for precise entailment scoring.
    Fallback: keyword overlap when NLI model not available.
    """
    sentences = [s.strip() for s in re.split(r'[.!?]+', answer) if s.strip()]
    if not sentences:
        return 0.0

    full_context = ' '.join(contexts)

    if nli_model:
        # Cross-encoder NLI: does context entail each claim?
        pairs  = [(full_context, s) for s in sentences]
        scores = nli_model.predict(pairs)
        # NLI labels: 0=contradiction, 1=neutral, 2=entailment
        supported = sum(1 for s in scores if s == 2)
    else:
        # Fallback: sentence-level embedding similarity
        sent_embs    = embedder.encode(sentences, normalize_embeddings=True)
        context_embs = embedder.encode([full_context], normalize_embeddings=True)
        sims         = (sent_embs @ context_embs.T).flatten()
        supported    = sum(1 for s in sims if s > 0.6)

    return supported / len(sentences)

# ── Answer relevance — does the answer address the question? ──────────
def answer_relevance_score(question: str, answer: str,
                             n_questions: int = 3,
                             llm_fn=None) -> float:
    """
    Generate n_questions from the answer, then measure how similar
    they are to the original question via embedding similarity.
    High similarity = answer is relevant to the original question.
    """
    if llm_fn:
        prompt = (
            f"Generate {n_questions} questions that this answer would answer.
"
            f"Answer: {answer}
Questions (one per line):"
        )
        generated_qs = llm_fn(prompt).strip().split('
')[:n_questions]
    else:
        # Simulate generated questions for demo
        generated_qs = [
            f"What is the process for {question.lower()}?",
            f"How does {question.lower().split()[0]} work?",
            question,
        ]

    q_emb  = embedder.encode([question],      normalize_embeddings=True)
    gq_emb = embedder.encode(generated_qs,    normalize_embeddings=True)
    sims   = (gq_emb @ q_emb.T).flatten()
    return float(sims.mean())

# ── Context recall — did retrieval find the right chunks? ─────────────
def context_recall_score(retrieved_contexts: list[str],
                          reference_answer: str) -> float:
    """
    Measure what fraction of reference answer sentences are
    supported by retrieved contexts. High score = retrieval is good.
    Low score = retrieval is the bottleneck.
    """
    ref_sentences = [s.strip() for s in re.split(r'[.!?]+', reference_answer)
                     if s.strip() and len(s.strip()) > 10]
    if not ref_sentences:
        return 0.0

    full_context = ' '.join(retrieved_contexts)
    ctx_emb      = embedder.encode([full_context], normalize_embeddings=True)
    sent_emb     = embedder.encode(ref_sentences, normalize_embeddings=True)
    sims         = (sent_emb @ ctx_emb.T).flatten()

    covered = sum(1 for s in sims if s > 0.55)
    return covered / len(ref_sentences)

# ── Evaluate a full RAG example ───────────────────────────────────────
rag_examples = [
    {
        'question':   "What does GATEWAY_ERROR mean on Stripe?",
        'retrieved':  ["Error code GATEWAY_ERROR indicates the bank gateway timed out. Retry the payment after 5 minutes."],
        'answer':     "GATEWAY_ERROR means the bank gateway timed out. You should retry the payment after 5 minutes.",
        'reference':  "GATEWAY_ERROR indicates the bank gateway timed out. Retry after 5 minutes.",
        'type':       'good — faithful and relevant',
    },
    {
        'question':   "What does GATEWAY_ERROR mean on Stripe?",
        'retrieved':  ["Error code GATEWAY_ERROR indicates the bank gateway timed out. Retry the payment after 5 minutes."],
        'answer':     "GATEWAY_ERROR typically means server overload. This is common during peak hours around 8-10 PM. Try switching payment methods or using a VPN.",
        'reference':  "GATEWAY_ERROR indicates the bank gateway timed out. Retry after 5 minutes.",
        'type':       'bad — hallucinated, unfaithful',
    },
    {
        'question':   "What is the international payment settlement time?",
        'retrieved':  ["Domestic payments settle within T+2 business days."],  # wrong chunk retrieved
        'answer':     "Payments settle within T+2 business days.",
        'reference':  "International payments settle within T+7 business days.",
        'type':       'bad — retrieval failed (context recall low)',
    },
]

print(f"{'Example':<10} {'Faithfulness':>14} {'Ans. Relevance':>16} {'Context Recall':>16} {'Type'}")
print("─" * 75)
for ex in rag_examples:
    f = faithfulness_score(ex['answer'], ex['retrieved'])
    a = answer_relevance_score(ex['question'], ex['answer'])
    c = context_recall_score(ex['retrieved'], ex['reference'])
    print(f"  {ex['type'][:8]:<10} {f:>14.3f} {a:>16.3f} {c:>16.3f}  {ex['type']}")

import numpy as np
from dataclasses import dataclass
from typing import Callable
import time

@dataclass
class RAGResult:
    answer:       str
    sources:      list[dict]
    faithfulness: float
    latency_ms:   float

class ProductionRAGPipeline:
    """
    Production-ready RAG pipeline with:
    - Hybrid search (dense + sparse + RRF)
    - Cross-encoder reranking
    - Citation tracking
    - Faithfulness checking
    - Latency monitoring
    """

    def __init__(self,
                  embedder,
                  reranker,
                  llm_fn: Callable[[str], str],
                  dense_index,
                  bm25_index,
                  chunks: list[dict]):
        self.embedder    = embedder
        self.reranker    = reranker
        self.llm_fn      = llm_fn
        self.dense_index = dense_index
        self.bm25        = bm25_index
        self.chunks      = chunks
        self.id_map      = {c['id']: c for c in chunks}

    def retrieve(self, query: str, k_coarse: int = 20,
                  k_final: int = 3) -> list[dict]:
        # Hybrid retrieval
        coarse   = hybrid_search(query, k_retrieve=k_coarse, k_return=k_coarse)
        # Cross-encoder reranking
        reranked = rerank(query, coarse, top_k=k_final)
        return reranked

    def build_prompt(self, query: str, chunks: list[dict]) -> str:
        context = '

'.join([
            f"[{i+1}] {c['text']}"
            for i, c in enumerate(chunks)
        ])
        return f"""You are a Stripe technical support assistant.
Answer the question using ONLY the context below.
After your answer, list which source numbers you used as [1], [2], etc.
If the answer is not in the context, say "I don't have that information."

Context:
{context}

Question: {query}
Answer:"""

    def check_faithfulness(self, answer: str, contexts: list[str]) -> float:
        return faithfulness_score(answer, contexts)

    def query(self, question: str) -> RAGResult:
        start = time.time()

        # Retrieve + rerank
        retrieved = self.retrieve(question, k_coarse=15, k_final=3)

        # Generate answer
        prompt = self.build_prompt(question, retrieved)
        answer = self.llm_fn(prompt)

        # Check faithfulness
        context_texts = [c['text'] for c in retrieved]
        faith = self.check_faithfulness(answer, context_texts)

        latency = (time.time() - start) * 1000
        return RAGResult(
            answer=answer,
            sources=retrieved,
            faithfulness=faith,
            latency_ms=latency,
        )

# ── Simulate pipeline usage ───────────────────────────────────────────
def mock_llm(prompt: str) -> str:
    """Mock LLM that returns a templated answer for demonstration."""
    if 'GATEWAY_ERROR' in prompt:
        return "GATEWAY_ERROR indicates the bank gateway timed out. Retry after 5 minutes. [1]"
    if 'settlement' in prompt.lower():
        return "Domestic payments settle in T+2 business days. International payments settle in T+7 days. [1][2]"
    return "I don't have that information in my knowledge base."

print("Production RAG pipeline queries:")
test_questions = [
    "What does GATEWAY_ERROR mean?",
    "How long does international settlement take?",
    "What is the meaning of life?",  # out of scope
]

for q in test_questions:
    retrieved = two_stage_retrieve(q, top_k_coarse=8, top_k_final=3)
    context_texts = [r['text'] for r in retrieved]
    prompt   = f"Answer based only on context:\n{chr(10).join(context_texts)}\n\nQ: {q}\nA:"
    answer   = mock_llm(q)
    faith    = faithfulness_score(answer, context_texts)

    print(f"
Q: {q}")
    print(f"A: {answer}")
    print(f"   Faithfulness: {faith:.3f}  Sources: {len(retrieved)}")

Slack message from support ops: "The assistant told a customer their refund would arrive in two to three days, but the actual policy for their payment method is five to seven. Someone needs to look at this today." Pulling the trace shows the retrieval step correctly found both the ACH refund timeline and the credit card refund timeline in its top chunks — context recall was fine. The reranker, trained on general web search data, scored the ACH chunk higher because it happened to share more surface wording with the question, even though the customer's payment method was a credit card. The fix is not "add more chunks" — it is tightening the prompt to require the model to identify the payment method before answering, and evaluating whether the reranker needs domain fine-tuning on exactly this kind of near-duplicate, policy-variant content.

Past a certain point, adding more retrieved chunks makes answers worse, not better. Every additional chunk is more text competing for the LLM's attention, and irrelevant or tangentially related chunks measurably increase the chance the model blends in something that does not belong in the answer — a direct hit to faithfulness. Retrieved context also costs money and latency per token whether or not it helped. The right target is the smallest set of chunks that actually contains the answer, which is exactly what reranking is for — narrowing a broad, high-recall candidate set down to a small, high-precision one, rather than simply raising the top-k number and hoping the LLM sorts it out.

Naive RAG is good enough for demos, where queries are hand-picked to work well, and for small, low-stakes internal tools. It reliably breaks on the failure modes this whole module exists to fix: missing exact keyword matches like error codes, retrieving chunks that are semantically similar but do not actually answer the question, and giving the team no objective way to know whether a change made things better or worse. The gap between a naive RAG demo and a production RAG system is not a matter of scale — it shows up on day one with any real, adversarial user base, which is exactly why hybrid search, reranking, and evaluation are treated as standard components, not optional extras, on any RAG system serving real users.

Chunk size and boundaries directly determine what can possibly be retrieved: a chunk boundary that splits a table in half, or separates an error code from its explanation, makes the correct answer unretrievable no matter how good the embedding model or reranker is downstream. Fixed-size chunking with no regard for document structure is the default because it is easy to implement, not because it performs best. Parent-child chunking, semantic or structure-aware splitting, and per-document-type chunking rules all exist because the "right" chunk size and boundary genuinely depends on the content — an FAQ entry, a legal contract clause, and an API reference table need different chunking logic to stay retrievable and coherent.

RAG reduces hallucination by giving the model relevant context to ground its answer in, but it does not force the model to use that context. As this module's error section covers, an LLM with strong prior knowledge on a topic will readily blend its own training-time beliefs with the retrieved context, or ignore the context outright, unless the prompt explicitly and strongly instructs it to answer only from what was retrieved. This is precisely why faithfulness is measured as its own separate metric from context recall — a RAG system can have perfect retrieval and still produce an unfaithful, hallucinated answer if the generation step is not held to the context.

They solve different stages of the same pipeline. Hybrid search is about recall at the first, cheap retrieval stage — making sure the correct chunk is somewhere in a reasonably sized candidate set by combining semantic and exact-keyword signals, since dense retrieval alone can miss exact terms and BM25 alone can miss paraphrases. Reranking is about precision at the second stage — given that broader candidate set, a cross-encoder scores each one far more accurately than either retrieval method could, to find the true top few. Skipping hybrid search means the correct chunk may never reach the reranker to be found in the first place; skipping reranking means a correct chunk that made it into the candidate set may still not surface at the very top.

A bi-encoder embeds the query and each document independently, which is what makes it fast enough to search millions of documents, but that independence means it can never model the specific interaction between a query term and a document term — it can only compare two fixed vectors. A cross-encoder reranker takes the query and a candidate document together as one input and scores their actual joint relevance, which is far more accurate but too slow to run over an entire corpus. Simply retrieving more documents with the bi-encoder does not fix this — it just hands the LLM a larger set of imprecisely ranked candidates. Reranking is what turns a fast, approximate top-100 into an accurate top-3.

A single retrieval call assumes the answer lives in one place, which breaks down on genuinely multi-hop questions. I would use query decomposition to split the question into sub-questions that are individually answerable, retrieve and answer each one separately, then have the LLM combine the sub-answers into a final response — or, for more open-ended multi-hop cases, run an iterative retrieve-then-read loop where the model reads an initial retrieval, decides what information is still missing, issues a follow-up retrieval for that gap, and repeats until it has enough to answer or explicitly reports that it cannot. Either approach needs its own evaluation, because standard single-hop context recall does not capture whether all the needed sub-facts were actually retrieved.

I would separate the pipeline into two things that can independently fail. Retrieval quality is measured with context recall — do the retrieved chunks actually contain the information needed to answer the question — which isolates whether the search and reranking stages are doing their job. Generation quality is measured with faithfulness — is every claim in the answer actually supported by the retrieved context — and answer relevance — does the answer address what was actually asked. Separating these matters for debugging: low context recall means the fix is in retrieval, chunking, or search strategy, while high context recall with low faithfulness means the fix is in the prompt or the generation step, not in retrieval at all.

I would start from the structure of the documents rather than picking an arbitrary token count — an FAQ entry, a paragraph of a policy document, and a row of an API reference table all have a natural unit that should not be split mid-thought. From there I would test a small range of chunk sizes against a hand-labelled evaluation set, measuring context recall at each size, since chunks that are too small lose surrounding context and chunks that are too large dilute relevance and hurt reranker precision. For content with an important internal hierarchy, like a document with sections and subsections, I would use parent-child chunking so retrieval can stay precise on small chunks while the LLM still receives the larger parent for full context.

I would explain that this trades one failure mode for a worse one: retrieving 20 chunks does raise the odds the right information is somewhere in the context, but it also means many more irrelevant or tangential chunks compete for the model's attention, which measurably increases hallucination risk and directly increases cost and latency on every single call. The fix for "we might be missing relevant chunks" is to widen the first-stage retrieval — increase the coarse candidate set to something like the top 20 or 50 — and then let a cross-encoder reranker narrow that back down to a precise top 3 for the actual prompt. That gets the safety margin the stakeholder wants without paying the attention-dilution and cost penalty of stuffing 20 chunks directly into the LLM.