RAG vs fine-tuning
RAG adds knowledge at question-time; fine-tuning adjusts behavior at training-time. The eternal interview question, settled by one distinction.
In plain terms
RAG is giving the employee the right reference binder for each task - swap pages anytime, always current. Fine-tuning is sending them to a training course - changes their style and habits permanently, but the course took weeks and its content ages. You don't send someone to a course to learn this morning's prices.
Why it matters
'Should we fine-tune on our documents?' is the most common expensive mistake in applied AI. Facts belong in retrieval (updatable, citable, access-controllable); persistent behaviors (format, tone, domain dialect) belong in tuning. Confusing them burns months.
How it works
Choose RAG when: knowledge changes, needs citations, differs per user/permission, or is large. Choose fine-tuning when: consistent style/format matters more than facts, few-shot prompts got long and repetitive, you need a small model to mimic a big one's behavior on a narrow task, or a domain dialect (medical, legal shorthand) confuses base models. They stack: fine-tune a model to follow RAG instructions and cite better, retrieval still supplies facts. Also: fine-tuning is bad at reliably adding facts - it mostly reshapes behavior.
When you use it
At architecture time on any project where someone (often a stakeholder) proposes fine-tuning. Default to RAG + prompting; earn fine-tuning with evidence.
Common mistakes
- Fine-tuning to inject facts, then rediscovering the model still hallucinates and the facts already changed.
- Underestimating tuning's ops burden: datasets, versioning, re-tuning on every base-model upgrade.
- Assuming they're rivals - the strongest systems use both, for different jobs.
Best practices
- Order of escalation: prompting -> few-shot -> RAG -> fine-tuning. Each step only when the previous measurably plateaus.
- Write the requirement down first: is the gap in what it knows or how it behaves? That sentence decides.
Try it yourself
For five product ideas (support bot, brand-voice writer, medical coder, docs Q&A, SQL generator), decide: RAG, fine-tuning, both, or neither - and defend each in one sentence.
Resources
- OpenAI - Optimizing LLMs (RAG vs FT framework) The escalation-ladder mental model, official version.