You Are the First Judge

Relax. We'll talk through this in plain words — no textbook voice.

What you'll walk away with

Understand why you're the first judge, with none of the intimidation
Get hands-on practice trying it yourself
Learn to smile and sidestep guesswork and secret leaks

Relax. We're not going to explain this chapter like a textbook — we'll just talk it through so it's easy to follow.

Let's think about it this way for a second

LLM-as-judge is a tool for later. For a beginner, reading it yourself is safer — because even judges can make things up. It's like having the server score the dish. Fine sometimes, but you don't need it for your very first dinner. You underline the musts, hunt for the must-nots, and mark Pass/Fail. As long as it takes under 5 minutes, this method is plenty.

Let's connect it to everyday life

When an answer comes back, read it carefully. Does each must show up? Does each must-not sneak in? If you're not sure, mark it Fail and write 'not sure' — don't hand out a Pass on a hunch. A courtroom doesn't give the benefit of the doubt. Only once you can score 5 cards yourself should you even think about an automated judge.

Read it yourself before you score it
When unsure, Fail + a note
Don't reach for a judge model yet

Let's try it together

text

Read the answer
Tick musts
Hunt must-nots
Unsure? → FAIL (note: unsure)
Only then save the card

You should see

You'll be able to mark an answer Pass/Fail yourself.

5-Minute Try-It

Write a sample answer with 2 musts and 1 must-not. Score it yourself and write down why.

A Quick Word of Caution

A judge who doesn't read the case turns the courtroom into theater. The score becomes decoration.

Easy traps

Marking Pass on 'it's probably fine' without actually reading it
Building a judge model on day one

Now Try It Yourself

Write a sample answer with 2 musts and 1 must-not. Score it yourself and write down why.

You'll know it worked when: You'll be able to mark an answer Pass/Fail yourself.