RAG Pipelines Essentials
> Retrieval-Augmented Generation (RAG) grounds an LLM's answers in your own data instead of relying solely on what it memorized during training. This notebook covers the core pipeline, the main design decisions, and a compact example of wiring one up.
What is RAG?
An LLM's knowledge is frozen at training time and limited to its context window. RAG works around both limits: instead of asking the model to answer from memory, you first retrieve relevant chunks of your own documents, then feed them into the prompt as context, and only then ask the model to generate an answer.
The result is a model that can answer questions about content it never saw during training — your internal docs, a codebase, last week's support tickets — with far less hallucination, because it's answering from text that's actually in front of it.
Why it matters:
Keeps answers grounded in your actual source of truth, not the model's parametric memory
Updates instantly — re-index a document instead of re-training a model
Enables citations — you know exactly which chunk of text produced the answer
Cheaper than fine-tuning for most "answer questions about our data" use cases
The Core Pipeline
Documents → Chunking → Embedding → Vector Store
↓
User Query → Embedding → Similarity Search → Retrieved Chunks
↓
Prompt = Query + Retrieved Chunks
↓
LLM → Answer
1. Ingestion (offline, done ahead of time)
Chunking — split documents into passages small enough to embed meaningfully and retrieve precisely (typically 200–1000 tokens, often with overlap)
Embedding — convert each chunk into a vector using an embedding model
Indexing — store vectors in a vector database alongside the original text and metadata
2. Retrieval (at query time)
Embed the user's query with the same embedding model
Run a similarity search (cosine similarity / nearest neighbor) against the vector store
Return the top-k most relevant chunks
3. Generation
Build a prompt that includes the retrieved chunks plus the user's question
Send it to the LLM and return the grounded answer
Common Use Cases
Internal knowledge assistants — answer employee questions from wikis, policy docs, runbooks
Customer support bots — ground responses in product documentation and past tickets
Codebase Q&A — retrieve relevant files or functions before answering "how does X work?"
Research assistants — retrieve from a corpus of papers or reports before synthesizing an answer
Compliance-sensitive domains — answers must be traceable to a specific source document
Practical Example: A Minimal RAG Pipeline
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
# 1. Ingestion — run once, or whenever source docs change
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(raw_documents)
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = Chroma.from_documents(chunks, embeddings)
# 2. Retrieval + 3. Generation — run per query
def answer_question(question: str) -> str:
retrieved = vector_store.similarity_search(question, k=4)
context = "\n\n".join(doc.page_content for doc in retrieved)
prompt = f"""Answer the question using only the context below.
If the context doesn't contain the answer, say you don't know.
Context:
{context}
Question: {question}"""
llm = ChatOpenAI(model="gpt-4o-mini")
return llm.invoke(prompt).content
print(answer_question("What is our refund policy for annual subscriptions?"))
This is deliberately minimal — a production pipeline usually adds re-ranking, metadata filtering, and citation tracking on top of this same shape.
Key Design Decisions
Chunk size — too small loses context, too large dilutes relevance and wastes tokens
Retrieval count (k) — more chunks give the model more context but increase cost and noise
Re-ranking — a second, more precise pass (often a cross-encoder) over the top-k results before they hit the prompt, common when the vector search alone isn't precise enough
Hybrid search — combining vector similarity with keyword search (BM25) catches cases pure embeddings miss, like exact IDs or acronyms
Metadata filtering — narrowing retrieval by source, date, or permissions before or during the similarity search
Common Pitfalls
Chunking without regard to structure — splitting mid-sentence or mid-table degrades retrieval quality
No citation/traceability — makes it impossible to verify or debug a wrong answer
Stale indexes — forgetting to re-embed when source documents change
Treating retrieval as "done" after launch — retrieval quality needs the same iteration and evaluation as the prompt itself
Key Takeaways
RAG separates "what the model knows" from "what's true right now" by retrieving fresh context at query time
The pipeline has two distinct phases — offline ingestion and online retrieval + generation
Retrieval quality, not prompt cleverness, is usually the biggest lever on answer quality
Start simple (chunk, embed, retrieve top-k) and add re-ranking or hybrid search only once you've measured a real gap
Resources
LangChain RAG docs: https://python.langchain.com/docs/tutorials/rag/
"Lost in the Middle" (Liu et al.) — on how context position affects retrieval usefulness