LLM Evaluation Guide
Scoring retrieval and generation separately - did we fetch the right chunks? did we answer faithfully from them? - because a single 'is it good?' hides which half is broken.
In plain terms
A RAG bot is a librarian plus a writer. When an answer is wrong, who failed - the librarian (fetched wrong pages) or the writer (had the right pages, botched the summary)? Grading them separately turns 'the bot is bad' into 'retrieval misses 30% of policy questions', which is fixable.
Why it matters
RAG has two failure surfaces multiplied together. Teams that only judge final answers thrash randomly between prompt tweaks and index rebuilds. Separated metrics point at the guilty stage - this is what makes RAG engineering rather than alchemy.
How it works
Retrieval metrics (no LLM needed): build 20-50 (question -> correct chunk) pairs; measure hit-rate@K (right chunk in top K?) and MRR (how high did it rank?). Generation metrics (LLM-as-judge over question + chunks + answer): faithfulness (every claim supported by the chunks?), relevance (actually answers the question?), plus 'correctly said I-don't-know when sources lacked the answer'. Tools like Ragas automate these, but hand-rolling the loop first teaches more.
When you use it
Before shipping any RAG feature, after every meaningful change (chunking, model, K, prompt), and continuously on sampled production traffic.
Common mistakes
- Eval questions written by re-reading the docs - real users ask vaguer, weirder things; harvest real queries ASAP.
- One blended score, hiding whether librarian or writer failed.
- Testing only answerable questions - the 'gracefully refuses' case IS a test case.
- Fixing a failure without adding it to the eval set (the same bug returns).
Best practices
- Retrieval eval is cheap and objective - automate it in CI like unit tests.
- Review 10 random production Q&A traces weekly; nothing replaces reading real transcripts.
- Track metrics per question-category to see which topics need better docs, not better code.
Try it yourself
Write 20 Q->chunk pairs for your project, including 3 unanswerable questions. Compute hit-rate@5. Now change ONE thing (chunk size, K, hybrid on/off) and re-measure. You are now doing real AI engineering.
Project: Phase project - cited docs-bot. A chatbot over a real document set (your college's rules, a product's docs, your notes): hybrid retrieval, inline numbered citations, honest 'not in my documents' refusals, a debug view showing retrieved chunks, and an eval report: hit-rate@5 plus faithfulness on 25 questions. This project is interview gold - most candidates have built RAG; almost none can show its eval report.
Deep dive
LLM evaluation is how you know whether a prompt, RAG system, or agent is actually improving instead of merely sounding better.
What it is
Evaluation compares model outputs against expected behavior, source evidence, user intent, and safety requirements.
Why it matters
Without evals, every prompt change is a guess. With evals, quality can improve without breaking old cases.
How it works
Create a small set of real cases, define pass criteria, run every change against the set, inspect failures, and add production failures back into the dataset.
Common mistakes
- Only testing happy paths.
- Using vague scores without examples.
- Evaluating RAG only by final answer.
Best practices
- Evaluate retrieval and generation separately.
- Keep a golden set of real user cases.
- Review traces weekly.
Resources
- Ragas Standard open-source RAG evaluation toolkit.
FAQ
How many eval cases do I need first?
Start with 20-30 real cases. Add more whenever production failures appear.
Should I use an LLM judge?
Use LLM judges for scale, but keep human-reviewed examples to calibrate them.