Before you can automate a check, you have to be able to say what "correct" means. A customer asked RefundBot: How long do I have to return my headphones? They were delivered last week. Below are ten replies it gave on different days.
Your task
Read the policy, then mark each reply Pass or Fail. A reply passes when everything it says agrees with the policy. Extra helpful detail is fine if it is true.
Write the rule you used in one sentence, so that another tester reading only your rule would mark the replies the same way.
Acme Store refund policy
Most items can be returned within 30 days of delivery for a full refund.
Refunds reach the original payment method within 5 to 7 business days of Acme receiving the item.
Gift cards cannot be refunded.
Damaged items: report within 48 hours of delivery with a photo, and pickup is free. Other return pickups cost Rs 99.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
1You can return them within 30 days of delivery, so you still have plenty of time.
2Returns are accepted for 14 days after delivery.
3Our return window is 30 days from the day your order is delivered. Refunds go back to your card within 5 to 7 business days.
4You have 30 days from the date you placed the order.
5Sorry, headphones can't be returned.
6Since they arrived last week, you're well inside the 30-day return window.
7You have 30 business days from delivery to return them.
8Items can be returned up to 30 days after delivery. Gift cards cannot be refunded.
9You can return them any time within the next 60 days.
10Yes! Returns are open for a month after delivery, which is 30 days.
How this is checked: Each row has a right label. At least 8 of 10 must match. For the full score, a harder check: all 10 must match.
2
Write it up
How this is checked: Not marked for content: it needs at least 8 words.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Writing a pass fail rule for chatbot replies that another tester would apply the same way
AI Testing for QA Engineers·Beginner·Why exact match fails on AI
How it works
Before anything can be automated, someone has to be able to say what "correct" means in a way another person would apply identically. For a support bot that usually means: every claim in the reply agrees with the written policy, and nothing the policy forbids is promised. Helpful extra detail is fine when it is true.
A rule written that way is the seed of every later test. A vague rule such as "the reply is accurate" lets two testers mark the same reply differently, and that disagreement will reappear later as a flaky check or a judge you cannot trust.
About the assignment
You read the refund policy, label ten real-looking RefundBot replies as pass or fail, and write the one-sentence rule you applied, so that a colleague reading only your rule would reach the same labels.
What you will do
1Read the policy panel first and note each fact it states: window, refund time, gift cards, damaged items, pickup fees.
2For each reply, check every claim against the policy, not just the main answer.
3Write your rule in terms of claims and the policy, not in terms of particular words.
Common mistakes
Failing a reply for being longer
Extra information is not wrong. A reply fails only when something in it disagrees with the policy.
Missing a small wrong claim
A reply can answer the main question correctly and still promise something false in its last sentence. Read to the end.
In a real test suite
Written pass criteria are what later become rubric questions, code checks and judge prompts. When a team argues about whether a reply is good, the cheapest fix is almost always a clearer rule written down before anyone looks at the next batch.
