A RAG system answers in two steps: it fetches documents, then writes an answer from them. A wrong answer can start in either step.
Why a tester cares
The fixes are different for each step, and some "failures" are not the system's fault at all: no document covers the question, or the expected answer was wrong.
Words you will meet
RAG:
retrieval-augmented generation: fetch relevant documents, then answer from them.
A RAG (retrieval-augmented generation) system answers in two steps: it retrieves documents, then generates an answer from them. A wrong answer can start in either step, and the fix is different: re-ranking or re-chunking for retrieval, prompt or model changes for generation.
Two more causes look like system failures and are not:
Content gap: no document covers the question. The fix is writing the document, not tuning the bot.
Bad gold: the bot is right and the expected answer in your eval set is wrong.
Your task
Each trace below shows the question, what was retrieved, the answer, and the eval set's expected answer. Decide where each failure starts.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
leave-2025: Full-time employees get 24 days of paid annual leave a year. Up to 10 unused days carry over to the next year. Leave must be approved by your manager in the HR portal.
leave-2023 (replaced): Full-time employees get 18 days of paid annual leave a year. Unused leave does not carry over. This policy was replaced on 1 April 2025.
wfh-2025: Employees may work from home up to 2 days a week with their manager's approval. New joiners work from the office for their first 30 days.
travel-2025: Book business travel through the travel portal at least 7 days ahead. Flights under 6 hours are booked in economy class.
expenses-2025: Submit expense claims within 30 days of the expense, with receipts. Meals on business travel are reimbursed up to Rs 1,500 a day.
notice-2025: The notice period is 60 days for permanent employees and 30 days during probation.
probation-2025: Probation lasts 6 months from the joining date. Your manager confirms you in the role at the end of it.
parental-2025: Employees get 26 weeks of paid maternity leave and 2 weeks of paid paternity leave.
holidays-2026: There are 12 fixed public holidays in 2026, plus 2 restricted holidays you choose from the HR portal list.
laptop-2025: Every employee gets a company laptop. Report a lost or stolen laptop to IT within 24 hours on extension 4040.
health-2025: Health insurance is a Rs 5 lakh family floater covering you, your spouse, two children and your parents.
referral-2025: The referral bonus is Rs 50,000, paid after the person you referred completes 90 days.
Twelve failing traces
R01How many days of annual leave do I get?
1. Retrieved1. leave-2023 (replaced): 18 days
2. leave-2025: 24 days
2. AnswerYou get 18 days of paid annual leave a year. [leave-2023]
3. Expected (eval set)24 days
R02How early must I book business travel?
1. Retrieved1. travel-2025: at least 7 days ahead
2. AnswerBook business travel at least 14 days ahead. [travel-2025]
3. Expected (eval set)7 days ahead
R03Do we get a gym membership?
1. Retrieved(no matching document)
2. AnswerI couldn't find that in Acme's HR policies.
3. Expected (eval set)Rs 2,000 a month
R04What is my notice period?
1. Retrieved1. notice-2025: 60 days, 30 during probation
2. AnswerThe notice period is 60 days, or 30 days during probation. [notice-2025]
3. Expected (eval set)90 days
R05How long is maternity leave?
1. Retrieved1. leave-2025: 24 days of annual leave
2. AnswerMaternity leave is 24 days. [leave-2025]
3. Expected (eval set)26 weeks
R06What is the referral bonus?
1. Retrieved1. referral-2025: Rs 50,000 after 90 days
2. AnswerThe referral bonus is Rs 5,000, paid after 90 days. [referral-2025]
3. Expected (eval set)Rs 50,000
R07Is there a company car scheme?
1. Retrieved(no matching document)
2. AnswerThere is no company car scheme in the policies I can see.
3. Expected (eval set)Yes, for L5 and above
R08How long is probation?
1. Retrieved1. probation-2025: 6 months
2. AnswerProbation lasts 6 months from your joining date. [probation-2025]
3. Expected (eval set)3 months
R09How many public holidays are there in 2026?
1. Retrieved1. holidays-2026: 12 fixed plus 2 restricted
2. AnswerThere are 14 fixed public holidays in 2026. [holidays-2026]
2. AnswerYour health cover is Rs 50,000. [referral-2025]
3. Expected (eval set)Rs 5 lakh family floater
R11Can I claim internet bills when working from home?
1. Retrieved1. wfh-2025: up to 2 days a week
2. expenses-2025: claims within 30 days
2. AnswerThe policies don't say whether internet bills can be claimed.
3. Expected (eval set)Up to Rs 1,000 a month
R12How long do I have to submit an expense claim?
1. Retrieved1. expenses-2025: within 30 days, with receipts
2. AnswerSubmit expense claims within 30 days, with receipts. [expenses-2025]
3. Expected (eval set)60 days
1
Label each one
Where does each failure start?
0 of 12 labelled
1R01: How many days of annual leave do I get?
2R02: How early must I book business travel?
3R03: Do we get a gym membership?
4R04: What is my notice period?
5R05: How long is maternity leave?
6R06: What is the referral bonus?
7R07: Is there a company car scheme?
8R08: How long is probation?
9R09: How many public holidays are there in 2026?
10R10: How much health cover do I get?
11R11: Can I claim internet bills when working from home?
12R12: How long do I have to submit an expense claim?
How this is checked: Each row has a right label. At least 9 of 12 must match. For the full score, a harder check: a few key rows must all be right.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
RAG testing: retrieval vs generation failures, content gaps and bad gold answers
AI Testing for QA Engineers·Intermediate·Testing RAG and hallucination
How it works
A retrieval-augmented generation system answers in two steps: it retrieves documents, then generates an answer from them. A wrong answer can start in either step, and the fixes differ: re-ranking or re-chunking for retrieval, prompt or model changes for generation.
Two other causes look like system failures and are not. A content gap means no document covers the question, so the fix is writing one. Bad gold means the bot was right and the expected answer in the eval set was wrong.
About the assignment
Read each trace's question, retrieved documents, answer and expected answer, and decide where each failure started.
What you will do
1Check first whether the right document was retrieved at all.
2If it was, check whether the answer follows it.
3Check the expected answer against the documents before blaming the bot.
Common mistakes
Blaming generation for everything
If the right document never came back, the answer had no chance.
Trusting the gold blindly
An expected answer can be the thing that is wrong.
In a real test suite
Separating the layers is what makes RAG bugs fixable. Teams measure retrieval with its own metrics, as the next lesson shows, before tuning prompts.
Labelling a batch of failures this way also produces a budget argument. If most failures start in retrieval, money spent on a bigger generation model is wasted; if many are content gaps, the fix belongs to the people who write the documents.
Disagreements between labellers about where a failure started are worth discussing. They usually mean a trace is missing a step, which is a logging problem to fix before the next round of analysis.
