RAG — Retrieval-Augmented Generation
Vector databases, semantic search, chunking strategies, and the full RAG pipeline from document to answer. Build a Stripe knowledge base Q&A system.
Fine-tuning teaches a model new behaviour. RAG gives a model access to documents it has never seen — without any training at all.
A customer asks Stripe's support bot: "What is the settlement cycle for international payments?" The LLM does not know — this is specific to Stripe's current policy which changes quarterly and was never in the training data. Fine-tuning would require retraining every time the policy changes. That is expensive, slow, and impractical.
RAG solves this differently. Before answering, it retrieves the most relevant sections from Stripe's documentation. Those sections are injected into the LLM's context window alongside the question. The LLM answers from the retrieved context — not from its weights. Update the documentation and the answers update instantly. No retraining. No fine-tuning.
RAG is now the standard architecture for any application that needs an LLM to answer questions about private, recent, or frequently-updated information. DoorDash's internal tool answering HR policy questions, Amazon's product Q&A bot, Brex's financial terms assistant — all are RAG systems.
An open-book exam vs a closed-book exam. Fine-tuning is memorising everything before the exam — works until the syllabus changes. RAG is the open-book exam — you bring the textbook and look up answers during the test. The student (LLM) still needs to be smart enough to find and synthesise the right information — but they do not need to memorise every fact in advance.
The retrieval step is critical — bringing the wrong textbook chapters into context produces wrong answers even with a perfect LLM. Most RAG failures are retrieval failures, not generation failures.
Two phases — indexing (offline) and retrieval+generation (online)
Chunking — how you split documents determines retrieval quality
Chunking is the single biggest lever in RAG quality. Too small: each chunk lacks context — the retrieved snippet is meaningless without surrounding text. Too large: the relevant sentence is buried in noise — the LLM hallucinates because it cannot find the answer in a 2000-token wall of text. The goal: each chunk should be semantically self-contained and contain exactly one answerable concept.
Split every N characters or tokens. Overlap of 10-20% between chunks.
Split on paragraphs first, then sentences, then words — trying to preserve semantic units.
Embed consecutive sentences. Split where embedding distance jumps — indicating a topic change.
Use headings, sections, and document structure to define chunks. Each section = one chunk.
FAISS, Chroma, and Pinecone — which vector database to use
A vector database stores embedding vectors and supports approximate nearest neighbour (ANN) search — finding the k most similar vectors to a query vector in milliseconds, even across millions of documents. Every RAG system uses one.
Complete RAG system — Stripe knowledge base Q&A
Grounding instruction prevents hallucination. Without it, the LLM will blend retrieved context with its own (potentially wrong) training knowledge.
RAG with OpenAI, Groq, and local models — production patterns
Every common RAG mistake — explained and fixed
Running RAG in production — re-indexing pipelines, embedding versions, and quality drift
The RAG pipeline built in this module runs once: load documents, embed, index, query. A production knowledge base changes constantly — new support articles, edited pricing pages, deprecated policies — and the index has to keep up without ever serving a query against a stale or partially-rebuilt index. The two problems that dominate a real RAG deployment are keeping the index fresh without downtime, and knowing when retrieval quality has quietly degraded, because nothing in the pipeline itself raises an alarm when it starts returning worse chunks.
Embedding model version mismatches are the single most common way a re-index pipeline silently breaks retrieval. Every chunk in the vector store has to be embedded by the exact same model version as the one embedding incoming queries — a cosine similarity computed between a vector from embedding model version A and a vector from version B is meaningless, even if both models have the same output dimension, because each model's vector space encodes similarity differently. Teams that upgrade an embedding model without re-embedding the entire existing index get a RAG system that still runs without errors and just quietly returns worse and worse retrieval results.
Live traffic monitoring closes the gap a fixed eval set cannot: real user queries are messier and more varied than any hand-built eval set, so production RAG systems also track the raw similarity score of the top retrieved chunk across all live queries, not just the labelled ones. A sustained drop in average top-1 score — even with no code change and no re-index — is often the first signal that the underlying document corpus has drifted away from what the index actually contains.
Five things people get wrong about RAG
RAG reduces hallucination by grounding generation in retrieved text, but it does not make hallucination structurally impossible. Two separate failure modes survive it: the LLM can still blend retrieved context with its own training-time priors and state something not actually in the context (the "LLM ignores retrieved context" error covered above), and if retrieval itself returns the wrong chunks, the model can generate a perfectly well-grounded, perfectly confident answer to the wrong information. Grounding instructions, temperature=0, and citation tracking narrow the gap; none of them close it completely.
They solve different problems and are frequently used together. RAG supplies fresh, private, or frequently-changing facts at query time without retraining — update the documents and the answers update instantly. Fine-tuning changes the model's behaviour: its output format, tone, task-following ability, or domain vocabulary — none of which retrieval can fix, because injecting more context does not teach a model to reliably output JSON or adopt a specific persona. A production support bot commonly does both: a fine-tuned (or well-prompted) model for consistent tone and format, fed retrieved context for the actual facts.
More context sounds strictly safer, but oversized chunks dilute the signal the retriever is trying to match on — a 2000-token chunk covering five topics gets a mediocre similarity score against a query about any one of them, and even when it is retrieved, the specific answer is buried in surrounding noise the LLM has to sift through. Chunking quality is about matching chunk boundaries to semantic units — one chunk, one answerable concept — not about maximising size. Overlap (10-20%) exists specifically to solve the opposite problem (a fact split across a chunk boundary), not as a reason to make every chunk larger.
This is exactly what a strong grounding instruction is trying to force, but it is not guaranteed behaviour — it is a tug-of-war between the prompt's instruction and the model's parametric priors, and the priors can win, especially on topics the model has strong, widely-corroborated training signal about. A model that "knows" a well-known public fact may still surface it even when a retrieved chunk explicitly states your organisation's different internal policy. This is precisely why production RAG systems test with adversarial queries where the retrieved context deliberately conflicts with common knowledge, and why citation tracking exists — to make it possible to catch, after the fact, when the model answered from memory instead of from the sources it was given.
Embedding similarity captures topical and semantic closeness well, but it is weak on exactly the things keyword/lexical search is strong at: exact identifiers, product SKUs, error codes, numbers, negation ("not eligible" embeds close to "eligible"), and rare domain jargon that the embedding model under-represents. This is why production retrieval systems increasingly use hybrid search — combining a sparse method like BM25 with dense vector search and merging the results (often with reciprocal rank fusion) — rather than relying on vector similarity alone. "Garbage in, garbage out" applies here directly: no amount of LLM quality compensates for a retriever that missed the one chunk with the exact number the user asked about.
RAG — 5 questions interviewers actually ask
Choose RAG when the model needs access to information that is private, large, or changes frequently — a knowledge base, product catalogue, or policy document — because retrieval lets you update the source documents and get updated answers with zero retraining. Choose fine-tuning when the problem is about the model's behaviour: getting it to reliably follow a specific output format, adopt a domain vocabulary, or perform a task pattern it does not do well zero-shot — none of which more context in the prompt reliably fixes. In practice the strongest systems combine both: a model fine-tuned (or carefully prompted) for consistent tone, format, and refusal behaviour, fed retrieved context at query time for the facts. I'd push back on a framing that treats them as alternatives — they operate on different axes of the problem.
Chunk size sets a trade-off between two failure modes on either side of it. Too small and a retrieved chunk lacks the surrounding context needed to make sense of it — you get back a sentence fragment that is technically similar to the query but unusable on its own. Too large and the chunk's embedding becomes an average over multiple unrelated ideas, which both hurts retrieval precision (the chunk scores lower against any single specific query) and, even when retrieved, buries the actual answer in surrounding text the LLM has to parse through. Overlap exists to solve a third, orthogonal failure: a single answerable fact sitting exactly on a chunk boundary and being split in half, so neither resulting chunk contains the complete answer. 10-20% overlap costs some storage and redundancy but meaningfully reduces boundary-split failures.
The LLM in a RAG system only sees what retrieval hands it — it has no independent channel back to the source documents to double-check or search further on its own. If the retriever returns the wrong chunks (wrong topic, outdated version, or simply missed the one relevant passage), the generation step is working from bad input no matter how capable the underlying model is; a stronger LLM will produce a more fluent and confident wrong answer, not a correct one. This is why, when a RAG system misbehaves, the very first debugging step should always be printing what was actually retrieved for that query, before looking at the generation step at all — in my experience the large majority of RAG failures are retrieval failures wearing a generation-failure costume.
There is no hard guarantee the model defers to the retrieved context — it is a probabilistic contest between the prompt's grounding instruction and whatever the model learned during pretraining, and strongly-held training priors (widely known facts) can win even against an explicit "use only the context below" instruction. You mitigate this at multiple levels: a strong, unambiguous grounding instruction and temperature=0 as the baseline; testing the system specifically with adversarial cases where retrieved context intentionally conflicts with common knowledge, to measure how often it fails; and citation tracking in production, so that when the model's claim doesn't match any cited source you can flag it automatically rather than trusting the answer at face value.
Retrieval, before generation, essentially every time. Print the actual chunks that were retrieved for the failing query and their similarity scores — often the answer is immediately visible: the right chunk wasn't in the top-k, the embedding model used for the query differs from the one used at indexing time, or the chunk exists but is missing the specific detail because of a bad chunk boundary. Only once you've confirmed the correct information was actually retrieved and handed to the model does it make sense to suspect the generation step — a weak grounding instruction, too high a temperature, or the model blending in its own priors. Debugging generation first, before verifying retrieval, is the most common wasted-effort mistake I see people make on RAG systems.
You can build a RAG system. Next: get better answers by engineering better prompts.
RAG handles the retrieval problem — getting relevant context into the LLM's window. But the quality of the generated answer also depends heavily on how the prompt is structured. Zero-shot, few-shot, chain-of-thought, ReAct — each prompting pattern consistently improves LLM outputs for different task types. Module 53 covers the patterns that actually work in production with real before/after examples.
Zero-shot, few-shot, chain-of-thought, ReAct — the patterns that consistently improve LLM outputs, with real before/after examples for every technique.
🎯 Key Takeaways
- ✓RAG gives an LLM access to documents it has never seen without any training. The pipeline has two phases: indexing (chunk documents → embed → store in vector DB, run once) and querying (embed question → vector search → retrieve top-k chunks → inject into LLM prompt → generate answer, runs on every request).
- ✓Chunking is the single biggest lever in RAG quality. Fixed-size chunking is simple but breaks semantic boundaries. Recursive character splitting is the practical default. Semantic chunking (split where embedding similarity drops) produces the best retrieval quality. Use 500–1000 tokens per chunk with 10–20% overlap to prevent key information from being split.
- ✓FAISS is the standard in-memory vector library for small-to-medium datasets. Chroma adds metadata filtering and automatic persistence. Pinecone is managed cloud for production scale. Always use the same embedding model and normalisation at index time and query time — mismatches silently produce wrong retrieval results.
- ✓The RAG prompt must include a strong grounding instruction: "Answer ONLY using the context below. If the answer is not in the context, say you do not have that information." Without this the LLM blends retrieved context with its own training knowledge and hallucinates. Set temperature=0 for factual Q&A.
- ✓Most RAG failures are retrieval failures not generation failures. If the LLM gives wrong answers, check what was retrieved first — print the top-k chunks. HyDE (Hypothetical Document Embeddings) improves retrieval for short or ambiguous queries: generate a hypothetical answer first, embed that, use it as the search vector.
- ✓Add citation tracking in production: number the retrieved chunks in the prompt and ask the LLM to cite which sources it used in its answer. This makes hallucination visible — if the LLM cites source [3] but source [3] does not contain the claimed fact, it hallucinated. Enables automatic fact-checking post-generation.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.