Write Three Eval Cards Yourself

Let's think about it this way for a sec

This practice set is here to put the basics from 'What is an eval', 'Pick the question', and 'Write a golden answer' into real hands-on use. Nothing new gets taught — you're just writing out concepts you already know by hand. Every eval card needs three things: the question, the golden/expected answer, and the pass/fail note. No dashboards, no leaderboards, no judge model required — you're the first judge here. By the end of this practice, you'll be able to write the shape of an eval card from muscle memory.

Exercises

Task 1 - Pick a simple FAQ question for a support chatbot and write a short golden answer. Task 2 - Write three pass/fail rules for that question (e.g. it must include the price, the tone must be polite, it must not mention a made-up feature). Task 3 - Add a trap case: write an edge-case question the model could easily get wrong, and note why it's a trap. Task 4 (optional, the harder one) - Take the three cards above, match each against a sample output, and decide pass/fail yourself.

Have you written a golden answer for each question
Are your pass/fail rules objective (checking facts/behavior, not style)
Have you included at least one trap case

Code Example

yaml

# eval-card-1.yaml
question: "Refund policy ဘယ်လိုလဲ?"
golden_answer: >
  Refund ကို purchase date ကနေ 30 ရက်အတွင်း
  receipt နဲ့တင်ရင် full refund ရပါတယ်။
pass_if:
  - mentions_30_day_window: true
  - mentions_receipt_required: true
  - no_invented_policy: true
fail_if:
  - promises_refund_without_receipt: true
notes: "Trap case - model က receipt မလိုဘူးလို့ မှားပြောတတ်တယ်"

# eval-card-2.yaml (သင့်ဘာသာ ဖြည့်ပါ)
question: ""
golden_answer: ""
pass_if: []
fail_if: []
notes: ""

You should see

You'll end up with three (or four) finished eval cards in a file, each pass/fail rule clear enough to check with a simple yes/no.

Try it in 5 minutes

Pick a support/FAQ question from an app you're using today, and write one eval card for it in 5 minutes.

A quick word of caution

Write every rule precisely enough to check with a plain yes/no — otherwise different judges (human or model) can land on different verdicts.