Exploring a new AI feature and reporting what is really broken is the core of the job. Here the faults sit in different layers, and some odd-looking behaviour is correct.
Why a tester cares
A useful bug report names the layer, says what happened, cites the trace that shows it, and says how often it happens. Reports about correct behaviour waste the team's time.
Words you will meet
Layer:
one part of the system: retrieval, generation or access control.
Reproduction rate:
how often the bug appears when you repeat the same question.
Trace ID:
the label of one logged request, so others can open it.
This PolicyPal build has seven planted defects across its layers, and it also does several things correctly that look odd at first. Your job is the one a tester does on a new AI feature: explore, reproduce, and report what is really broken, with evidence.
Every console reply shows its trace ID. A good report names the failing layer (retrieval, generation, access or behaviour), says what happened, cites the trace that shows it, and gives a reproduction rate.
Your task
Ask the console about PolicyPal's documents as a Sales employee. Then write your defect reports as a JSON list:
[
{"title": "...", "layer": "retrieval", "evidence": "tr-1a2b3c4d: answer said ...", "reproduction": "3 of 3"}
]
Reports about correct behaviour count against you. Check the documents before reporting.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
leave-2025: Full-time employees get 24 days of paid annual leave a year. Up to 10 unused days carry over to the next year. Leave must be approved by your manager in the HR portal.
leave-2023 (replaced): Full-time employees get 18 days of paid annual leave a year. Unused leave does not carry over. This policy was replaced on 1 April 2025.
wfh-2025: Employees may work from home up to 2 days a week with their manager's approval. New joiners work from the office for their first 30 days.
travel-2025: Book business travel through the travel portal at least 7 days ahead. Flights under 6 hours are booked in economy class.
expenses-2025: Submit expense claims within 30 days of the expense, with receipts. Meals on business travel are reimbursed up to Rs 1,500 a day.
notice-2025: The notice period is 60 days for permanent employees and 30 days during probation.
probation-2025: Probation lasts 6 months from the joining date. Your manager confirms you in the role at the end of it.
parental-2025: Employees get 26 weeks of paid maternity leave and 2 weeks of paid paternity leave.
holidays-2026: There are 12 fixed public holidays in 2026, plus 2 restricted holidays you choose from the HR portal list.
laptop-2025: Every employee gets a company laptop. Report a lost or stolen laptop to IT within 24 hours on extension 4040.
health-2025: Health insurance is a Rs 5 lakh family floater covering you, your spouse, two children and your parents.
referral-2025: The referral bonus is Rs 50,000, paid after the person you referred completes 90 days.
Restricted: eng-bands-2025 (Engineering managers only).
1
PolicyPal (you are in Sales)
No messages yet. Start with a suggested question below.
2
Write it up
How this is checked: When you submit, each report is matched with the faults planted in your copy of the product, and each must cite a trace ID from your own log. Reports that match nothing count against you.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Debug a RAG chatbot and write defect reports with the failing layer and a traced reproduction
AI Testing for QA Engineers·Advanced·Testing RAG and hallucination
How it works
Exploring a new AI feature and reporting what is really broken is the core job. Planted defects here sit in different layers, and some correct behaviour looks odd at first, so careful checking matters as much as finding things.
A useful report names the failing layer, says what happened, cites the trace that shows it, and gives a reproduction rate. Reports about correct behaviour cost the team time and count against you.
About the assignment
Explore PolicyPal as a Sales employee and write a JSON list of defect reports with layer, evidence and reproduction rate.
What you will do
1Ask about every topic in the documents, and some they do not cover.
2Compare each answer and citation with the documents panel.
3Write one report per defect, citing a trace from your own console log.
Common mistakes
Evidence without a trace
A report must point at a call in your own log, so others can reproduce it.
Reporting correct behaviour
Check the documents before reporting. Some odd-looking answers are right.
In a real test suite
Good defect reports are the most visible output of an AI tester. Capstone 1 repeats this exercise on a build generated for you, alongside a full evaluation report.
Give reproduction rates even when they are 3 of 3. A defect that appears once in ten tries needs a different fix and a different test from one that appears every time, and the rate is what tells the developer which they are dealing with.
