Phase 06 · RAG - retrieval-augmented generation Core

What is RAG & why it exists

RAG = retrieve relevant documents for the user's question, paste them into the prompt, and instruct the model to answer only from them.

In plain terms

A vanilla LLM answers from memory like a professor with no library card - fluent, but frozen in time and unable to see your company's files. RAG hands the professor the three most relevant pages before they answer, and tells them to quote their sources. Open-book exam instead of closed-book.

Why it matters

Three unsolvable problems of bare LLMs, solved at once: (1) knowledge cutoff - models know nothing after training day; (2) private data - they've never seen your wiki, contracts or tickets; (3) hallucination - grounding answers in provided text with citations makes wrongness rarer and detectable. That trifecta is why 'chat with your docs' is the most-built AI product in existence.

How it works

Two phases. Offline (ingestion): documents -> chunks -> embeddings -> vector index (exactly your Phase-04 project). Online (per query): embed the question -> fetch top-K chunks -> assemble prompt ('Answer using only these sources; cite them; if the answer isn't here, say so') -> generate -> show answer + citations.

When you use it

Knowledge that changes, private corpora, anything requiring source-of-truth answers: support bots, internal wikis, legal/medical assistants, documentation Q&A. Not needed for pure reasoning/creative tasks with no external facts.

Common mistakes

  • Thinking RAG eliminates hallucination - it reduces it; the model can still misread or blend sources, hence evals.
  • Skipping the 'if not in sources, say I don't know' instruction - the single highest-value line in the prompt.
  • Fine-tuning to 'teach the model our docs' when RAG is cheaper, fresher and auditable (-> RAG vs fine-tuning).

Best practices

  • Always show citations - trust and debuggability in one feature.
  • Keep the index fresh; stale RAG confidently serves last year's policy.
  • Log question, retrieved chunks AND answer - you can't debug what you didn't capture.

Try it yourself

Manually simulate RAG once: take a question, personally find the right paragraph in a document, paste both into a chat model with 'answer only from this excerpt, cite it'. Then ask the same question without the excerpt. That contrast is the entire pitch.

Resources

Related topics

Build: semantic search end-to-end The rite-of-passage build: ingest documents -> chunk -> embed -> index -> query - the complete retrieval pipeline that RAG will sit on. The RAG pipeline, end to end Wiring ingestion and query-time retrieval into one system, where overall quality is the product of every stage's quality. RAG vs fine-tuning RAG adds knowledge at question-time; fine-tuning adjusts behavior at training-time. The eternal interview question, settled by one distinction. Context window The maximum number of tokens a model can consider at once - its entire working memory, including your prompt, the history, and its answer.