Your First Eval, on Paper

Relax. We'll talk through this in plain words — no textbook voice.

What you'll walk away with

Understand writing your first eval on paper, so it stops feeling intimidating
Get hands-on practice trying it yourself
Learn to spot and sidestep guesswork and secrets with a smile

Take a breath. Let's break this chapter down like a conversation, not a textbook, so it's easy to follow.

Let's think about it this way for a sec

A paper eval means there's no UI yet. You write it like a card — a name, the question, must / must-not, when to rerun it, which prompt/context it pairs with. If you want it to connect with a prompt pack or context pack, give it a matching name. If there's a hidden 'magic box' expectation in your head, pull it out as an explicit must. Only try it in a real chat once the card itself is clear.

Let's connect it to everyday life

Sample card — Name: Cite the handbook. Ask: Am I allowed to use my phone? Must: 'No,' handbook.md. Must not: 'Allowed.' When: after editing the prompt, after editing the file. Keep your first pack to 5 cards or fewer. You don't need a leaderboard yet.

Every card has ask + must + must-not
Note when it should be rerun
Name it after the behavior it checks

Let's try it together, hands-on

text

Card: Cite the handbook
Ask  → Must / Must not
When → after prompt or note change
Then run. Write PASS/FAIL + one line.

You should see

You'll be able to write an eval card on paper.

Try it in 5 minutes

For 3 cards, write the name, question, must, must-not, and when to use each.

A quick word of caution

If you can't read the must on the card clearly out loud yet, don't try running it yet.

Easy traps

Opening the chat first instead of writing the card
Naming cards 'Test 1' / 'Test 2'

Now try it yourself

For 3 cards, write the name, question, must, must-not, and when to use each.

You'll know it worked when: You'll be able to write an eval card on paper.