Don't Let the Model Grade Itself

Relax. We're not going to explain this chapter like a textbook — we'll just talk it through so it's easy to follow.

Let's think about it this way for a second

The RAG and Prompt Engineering lessons already warned you: a model can be confidently wrong. Ask it to grade its own answer, and that same confidence gets rewarded. 'Is this a 10 out of 10?', 'Is this correct?', 'Grade this honestly' — these are traps. It's like asking a friend who always says yes. Better: you check it yourself. Second-best: check it with a different card, on a different day, against a different set of notes.

Let's connect it to everyday life

Bad — trusting a 9 out of 10 the model gave itself under 'Rate your confidence 1–10.' Good — checking it off against your own must list. Second card — a question with no answer in the source. Third card — a trap question. Only once all three pass should you call the behavior stable. A lone 9/10 doesn't mean stable.

• Don't let it grade its own answer
• Don't treat a confidence number as proof
• Check stability across three cards

Let's try it together

Don't ask: “Are you sure? Score yourself.”
Do:        check musts yourself
            run the missing-note card
            run the trap card

You'll be able to explain why letting it grade itself isn't an eval.