Phase 06 · RAG - retrieval-augmented generation Core

RAG Pipeline Explained

Wiring ingestion and query-time retrieval into one system, where overall quality is the product of every stage's quality.

In plain terms

RAG is an assembly line: parse -> chunk -> embed -> index -> retrieve -> assemble prompt -> generate -> cite. Like any assembly line, the worst station sets your ceiling - a perfect prompt cannot rescue retrieval that fetched the wrong page, and perfect retrieval can't fix a parser that mangled the PDF.

Why it matters

Beginners obsess over the generation prompt because it's visible, but production RAG failures are overwhelmingly retrieval failures - and retrieval failures are usually parsing and chunking failures. Knowing the whole chain tells you where to look.

How it works

Stages and their classic failure: Parsing (PDFs are hostile; tables and layouts mangle - inspect extracted text!) -> Chunking (-> its node) -> Embedding -> Indexing (metadata for filters) -> Query handling (users type vague fragments; consider rewriting the query with an LLM first) -> Retrieval (top-K, filters) -> Prompt assembly (sources labeled [1],[2], grounding rules) -> Generation (low temperature) -> Citation display. Debug by inspecting the artifact between each pair of stages.

When you use it

This is the blueprint for every document-QA build, and the checklist for debugging one.

Common mistakes

  • Debugging the prompt when retrieval returned garbage - ALWAYS print retrieved chunks first; 80% of the time your answer is there (or visibly isn't).
  • Trusting PDF extraction blindly (silent table destruction is the norm).
  • Fixed K=5 for every query - some need one chunk, some need twelve.
  • Letting the framework assemble prompts you've never actually read.

Best practices

  • Build a 'debug view' showing: raw query -> rewritten query -> top chunks with scores -> final prompt -> answer. Non-negotiable tooling.
  • Evaluate retrieval separately from generation: 'was the right chunk in top-K?' is measurable without any LLM judge.
  • Start raw (your Phase-04 code + one prompt); add frameworks only when you feel the pain they solve.

Try it yourself

Extend your Phase-04 search project into full RAG: add the grounded-answer prompt with citations. Then break it on purpose - ask something NOT in the docs and verify it says 'I don't know' rather than inventing.

Deep dive

A RAG pipeline gives an AI app the right context before it answers. Instead of relying on model memory, the system retrieves relevant documents and uses them to generate grounded responses.

What it is

RAG means retrieval-augmented generation. It combines search with an LLM so answers can use private, fresh, or domain-specific information.

Why it matters

Most useful AI products need facts outside the model. RAG reduces hallucination, enables citations, and lets teams update knowledge without retraining a model.

How it works

Load documents, split them into chunks, embed the chunks, store vectors, retrieve top matches for a query, place those matches into a prompt, then generate and evaluate the answer.

Common mistakes

  • Chunking documents blindly by fixed character count.
  • Only checking final answers instead of retrieved chunks.
  • Ignoring metadata filters for permissions and categories.

Best practices

  • Inspect retrieved chunks for every failure.
  • Track retrieval hit rate separately from answer quality.
  • Start simple before adding reranking or complex agents.

Practical workflow

query -> retrieve relevant chunks -> build prompt with sources -> generate answer -> evaluate citations

Resources

FAQ

Is RAG better than fine-tuning?

Use RAG when the model needs external knowledge. Use fine-tuning when you need behavior, style, or task adaptation.

Do I need a vector database for RAG?

For production, usually yes. For small demos, local vector stores or simple semantic search can be enough.

Related topics

What is RAG & why it exists RAG = retrieve relevant documents for the user's question, paste them into the prompt, and instruct the model to answer only from them. Chunking strategies Splitting documents into right-sized pieces before embedding - the humble step that quietly decides retrieval quality more than any model choice. Hybrid search & reranking Combine semantic search with old-school keyword search (hybrid), then have a smarter model re-order the candidates (reranking) - the two standard upgrades when plain RAG plateaus. Evaluating & debugging RAG Scoring retrieval and generation separately - did we fetch the right chunks? did we answer faithfully from them? - because a single 'is it good?' hides which half is broken. Prompt engineering vs context engineering Prompt engineering crafts the instructions; context engineering builds the systems that decide what information fills the model's window at runtime. The NLP pipeline, end to end The sequence of steps that turns raw text into something a model can actually use. Vector databases Databases purpose-built to store millions of embeddings and answer 'find the K nearest vectors to this one' in milliseconds. Build: semantic search end-to-end The rite-of-passage build: ingest documents -> chunk -> embed -> index -> query - the complete retrieval pipeline that RAG will sit on. Documents, loaders and text splitters A Document is text plus Metadata, usually created by a Document Loader and broken into Chunk objects by a Text Splitter. Retrievers and RAG chains A Retriever turns a query into relevant Source Documents using embeddings, vector search and ranking strategies. Capstone production AI product One deployed project that combines retrieval, tools, evals, security, cost tracking, and documentation.