Build a Study Eval Pack

Relax. We'll talk through this in plain words — no textbook voice.

What you'll walk away with

Understand building a study eval pack, so it stops feeling intimidating
Get hands-on practice trying it yourself
Learn to spot and sidestep guesswork and secrets with a smile

Take a breath. Let's break this chapter down like a conversation, not a textbook, so it's easy to follow.

Let's think about it this way for a sec

This project isn't anything new. Happy / missing / trap, must / must-not, being your own judge, rerunning after every edit, keeping secrets out, writing coverage — it all lands in one pack. The cards are: Cite the notes, Stop if missing, No fortune telling, Keep the shape, Don't leak secrets. One behavior per card. Then edit a prompt and call three cards back for a rerun. That's what makes the pack truly yours.

Let's connect it to everyday life

Write it all in one file. Every card gets When, Ask, Must, Must not. Tested / Untested at the top. v1 / v2 in eval-log.md. If you already have a prompt pack and context pack, link the names together. You don't need a UI yet. You don't need a judge model yet. Once your little card deck is done, the course is done. You can always add more cards later.

Write 5 cards
Edit one thing and rerun all three cards
Attach coverage notes and a log

Let's try it together, hands-on

markdown

# Study Eval Pack

1. Cite the notes
2. Stop if missing
3. No fortune telling
4. Keep the shape
5. Don’t leak secrets

Each card: When + Ask + Must + Must not
Log: v1 → v2 after one prompt change

You should see

You'll finish a tested Study Eval Pack.

Try it in 5 minutes

Write 5 cards. Edit one line of the prompt and rerun 3 cards. Attach the log and coverage notes.

A quick word of caution

A pretty pack doesn't mean the bot is correct. It only becomes your real tool once you've watched it fail after an edit.

Easy traps

Writing 20 cards and never running a single one
Not keeping a log, so you can't tell what changed

Now try it yourself

Write 5 cards. Edit one line of the prompt and rerun 3 cards. Attach the log and coverage notes.

You'll know it worked when: You'll finish a tested Study Eval Pack.