How Do You Test a RAG?

A RAG being "done" doesn't mean the UI looks nice. It's done when it can clear three kinds of questions: the correct question, the missing question, and the question tempting it to make something up.

Let's think about this for a second

Testing is your exam. Happy path — a question that's actually in the notes, and the source needs to be correct. Missing — not in the notes, and it needs to say so. Temptation — a question baiting it to add scientific-sounding detail it shouldn't make up. Only add more files once you've passed all three. Shipping after testing just one demo question is like getting a degree without ever sitting the exam.

Let's connect it to everyday life

Test 1 — "Can I use my phone?" → phones.md, the quote needs to be accurate. Test 2 — "canteen menu" → not found. Test 3 — "the chemical formula for chlorophyll" — if it's not in the notes, it shouldn't be added. Test it by hand yourself first, then automate it with a script. Turn these three into checkboxes in your Definition of Done.

Test one happy-path question
Test one missing-info path
Shut down a question tempting it to make something up

Let's try it hands-on together

text

Definition of done
[ ] known question → correct file + quote
[ ] unknown question → “မတွေ့”
[ ] no invented facts
[ ] no web search
[ ] secrets refused
[ ] I can explain the search step

You should see

You'll be able to demonstrate three tests that determine whether a RAG is truly done.

5-minute try-it

Write 3 test cards for your own RAG. Include the input, the expected file, and the expected answer.

One quick warning

If you only test with the demo question, it'll fall apart on the very first question a friend actually asks.