Not every AI feature has documents to check against. Then you check an answer the way an editor would: split it into claims and check each against a trusted reference.
Why a tester cares
Most hallucinations that reach users are claims the reference simply does not cover, stated confidently.
Not every AI feature has documents to check against. For factual questions and summaries, the method is the same as a careful editor's: break the answer into claims, then check each against a trusted reference.
Supported: the reference says it, in any wording.
Contradicted: the reference says something else.
Not in the reference: it might be true, but nothing here confirms it. Stated confidently, this is how most hallucinations reach users.
The eight answers below were written for this course in the style of real model answers, each paired with a reference.
Your task
Label every claim.
For the claims the reference does not cover, say whether the answer stated them confidently or hedged them.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
A1When did India's Digital Personal Data Protection Act become law?
1. ReferenceThe Digital Personal Data Protection Act, 2023 received the President's assent on 11 August 2023.
2. Claim A1aThe Act is from 2023.
3. Claim A1bIt received assent on 11 August 2023.
4. Claim A1cIt was passed unanimously by both houses.
A2What does the Wilson interval give for 18 of 20?
1. ReferenceFor 18 passes out of 20 at 95% confidence, the Wilson score interval is about 69.9% to 97.2%.
2. Claim A2aThe interval runs from about 70% to 97%.
3. Claim A2bThe interval is exactly 85% to 95%.
A3What is Acme's return window?
1. ReferenceMost items can be returned within 30 days of delivery for a full refund.
2. Claim A3aYou have a month, thirty days from the day it arrives, to send most things back.
3. Claim A3bOpened electronics have a 15-day window.
A4How long is paid maternity leave at Acme Corp?
1. ReferenceEmployees get 26 weeks of paid maternity leave and 2 weeks of paid paternity leave.
2. Claim A4aMaternity leave is 26 weeks, fully paid.
3. Claim A4bPaternity leave is 4 weeks.
A5Who covers the Rs 99 pickup fee for damaged items?
1. ReferenceDamaged items: report within 48 hours of delivery with a photo, and pickup is free. Other return pickups cost Rs 99.
2. Claim A5aDamaged items are picked up free of charge.
3. Claim A5bThe Rs 99 fee applies to every return.
4. Claim A5cPickups are usually made the next working day.
A6What is McNemar's test used for?
1. ReferenceMcNemar's test compares two paired proportions, using only the discordant pairs.
2. Claim A6aIt compares two versions on the same cases.
3. Claim A6bIt uses every case, including those both versions passed.
A7What is the notice period during probation?
1. ReferenceThe notice period is 60 days for permanent employees and 30 days during probation.
2. Claim A7aDuring probation, either side gives 30 days' notice.
3. Claim A7bMost companies in India use a 90-day notice period.
A8What is pass^k?
1. Referencepass^k is the probability that all k independent attempts at a task succeed.
2. Claim A8apass^k is the chance that every one of k tries succeeds.
3. Claim A8bpass^k is the chance that at least one of k tries succeeds.
4. Claim A8cIt was first defined in 2021.
1
Label each one
Each claim against its reference
0 of 19 labelled
1A1The Act is from 2023.
2A1It received assent on 11 August 2023.
3A1It was passed unanimously by both houses.
4A2The interval runs from about 70% to 97%.
5A2The interval is exactly 85% to 95%.
6A3You have a month, thirty days from the day it arrives, to send most things back.
7A3Opened electronics have a 15-day window.
8A4Maternity leave is 26 weeks, fully paid.
9A4Paternity leave is 4 weeks.
10A5Damaged items are picked up free of charge.
11A5The Rs 99 fee applies to every return.
12A5Pickups are usually made the next working day.
13A6It compares two versions on the same cases.
14A6It uses every case, including those both versions passed.
15A7During probation, either side gives 30 days' notice.
16A7Most companies in India use a 90-day notice period.
17A8pass^k is the chance that every one of k tries succeeds.
18A8pass^k is the chance that at least one of k tries succeeds.
19A8It was first defined in 2021.
How this is checked: Each row has a right label. At least 16 of 19 must match. For the full score, a harder check: a few key rows must all be right.
2
Label each one
The claims the references do not cover: how were they worded?
0 of 5 labelled
1A1c: It was passed unanimously by both houses.
2A3b: Opened electronics have a 15-day window.
3A5c: Pickups are usually made the next working day.
4A7b: Most companies in India use a 90-day notice period.
5A8c: It was first defined in 2021.
How this is checked: Each row has a right label. All 5 must match. This is a harder check: it earns the second 50 points once the core checks pass.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Detect hallucinations by checking claims against a reference: supported, contradicted or unverifiable
AI Testing for QA Engineers·Intermediate·Testing RAG and hallucination
How it works
Not every AI feature has documents to check against. For factual questions and summaries, the method is a careful editor's: break the answer into claims, then check each against a trusted reference.
Each claim is supported, contradicted, or not covered by the reference. The last group is where most hallucinations reach users, especially when the claim is stated confidently, because nothing contradicts it and the tone invites trust.
About the assignment
Label every claim in eight answers against their references, and for claims the reference does not cover, judge whether the wording was confident.
What you will do
1Split each answer into separate factual claims.
2Find each claim in the reference, in any wording.
3For uncovered claims, judge whether they were hedged or stated as fact.
Common mistakes
Marking uncovered claims as wrong
They may be true. What matters is that nothing here confirms them.
Treating paraphrase as contradiction
Different words for the same fact still support it.
In a real test suite
Claim-level checking is slow by hand, which is why teams use it on samples to build a picture of how a model fails, and then automate the parts code or a judge can do.
Keep the references you check against under version control too. A summary checked against an outdated reference will show false contradictions, and those make the method look unreliable when the real problem is the reference.
Count the uncovered claims per answer as well as labelling them. A rising count after a model change is an early sign that the model has become more willing to fill gaps, even when no single claim is clearly wrong.
