Pick the Question

Relax. We'll talk through this in plain words — no textbook voice.

What you'll walk away with

Understand how to pick the question to test, with none of the intimidation
Get hands-on practice trying it yourself
Learn to smile and sidestep guesswork and secret leaks

Relax. We're not going to explain this chapter like a textbook — we'll just talk it through so it's easy to follow.

Let's think about it this way for a second

If your question is too broad, you can't tell pass from fail. 'Is the AI good?' isn't a question. 'Per the handbook, is phone use allowed?' is a question. You need to be able to write down the correct behavior in advance. If you can't, what you have is an exploratory chat, not yet an eval. Don't ask the server 'was the food good?' Ask 'was the fish dish not spicy?'

Let's connect it to everyday life

Good — 'Per the Week 1 notes, why does a plant need sunlight? Two sentences.' Bad — 'Explain photosynthesis fully, with an example, a question, and a poem.' The second one mixes three tasks together — you won't know which part failed. In your first pack, one question = one behavior. Same idea as the Prompt Engineering lesson: one task at a time.

One question = one behavior
You must be able to write the expectation in advance
Don't turn 'is it good?' into your question

Let's try it together

text

Too wide: Is my study bot good?
Better:   From week-1 notes, why do plants need sun?
Expect:   2 sentences + no exam jargon

You should see

You'll be able to pick a short, focused question for an eval.

5-Minute Try-It

Write 2 overly broad questions. Split each one into 2 short, focused questions.

A Quick Word of Caution

Don't eval a question you don't know the answer to yourself. That's just you guessing, then grading it as if you knew.

Easy traps

Cramming 5 tasks into one question
Turning a question you can't answer yourself into an eval

Now Try It Yourself

Write 2 overly broad questions. Split each one into 2 short, focused questions.

You'll know it worked when: You'll be able to pick a short, focused question for an eval.