Secrets Hiding in Your Test Notes

Relax. We'll talk through this in plain words — no textbook voice.

What you'll walk away with

Understand secrets hiding in test notes, so it stops feeling intimidating
Get hands-on practice trying it yourself
Learn to spot and sidestep guesswork and secrets with a smile

Take a breath. Let's break this chapter down like a conversation, not a textbook, so it's easy to follow.

Let's think about it this way for a sec

Test notes spread further than your context does. You'll want to share them, rerun them, screenshot them. So use fake names, fake dates, fake files. The RAG/Prompt/Context lessons already warned you about secrets. The extra danger with evals is writing a real key into a must-not rule, like 'this text must not appear' — that still means you typed the actual key into the file.

Let's connect it to everyday life

Safe to use — Mina, handbook-fake.md, due Friday. Not safe — sk-live, real student grades, an actual phone number. For a trap card, use a fake secret, like 'API_KEY=demo-not-real.' Never put a real key in. And read through eval-log.md before you commit it.

Use fake text in golden answers
Never write a real key as a must-not rule
Before sharing, read through the file first

Let's try it together, hands-on

text

Safe trap:   “demo-not-real” must not appear
Unsafe trap: paste a live key and say “don’t repeat this”

You should see

You'll be able to identify secrets that should never go into an eval file.

Try it in 5 minutes

Write 4 sample cards. Split them into 2 safe and 2 unsafe, and note why.

A quick word of caution

If you tell the model 'don't repeat this' and then hand it a real key, that's not an eval — that's a leak.

Easy traps

Writing a real key inside a must-not rule
Sharing screenshots without blanking out the numbers in them

Now try it yourself

Write 4 sample cards. Split them into 2 safe and 2 unsafe, and note why.

You'll know it worked when: You'll be able to identify secrets that should never go into an eval file.