Skip to content
CodeKruPro

Real failure hunt

61/99

Module · Testing RAG and hallucination

advanced

about 60 min

What this lesson teaches

The idea

Planted faults teach the shape of failures, but real models fail in ways nobody planted. Here you read a real model's answers against the documents it was given.

Why a tester cares

Real failures are often subtle, such as agreeing with a wrong number the user suggested. Each finding becomes a regression test so it cannot come back unnoticed.

Words you will meet

Finding:
one recorded failure, with the trace that shows it.
Regression test:
a test that fails if a fixed bug comes back.
Mock:
a stand-in version of the product that tests can run against cheaply.

What you will do

  1. 1Send the 30 questions
  2. 2Find three real failures
  3. 3Label them
  4. 4Write regression tests
Jump to practice

PolicyPal's planted defects are useful because you know they are there. Real models fail in ways nobody planted. In this lesson the console sends questions to a real model, with PolicyPal's documents as its context, and your job is to find real failures in what comes back.

Real failures are often subtle: agreeing with a wrong number the user suggested, a confident answer about a topic no document covers, a citation to a document that only half supports the sentence.

Your task

  1. Send the 30 questions (one click sends them all) and read every reply against the documents.
  2. Record three findings as JSON, each with the reply's trace ID and a category from the list. A finding is a real failure. If the model got a tricky question right, such as one that pushes a wrong number, record it as held_up and say what you checked: a hunt that finds little is still a result.
  3. Turn each failure into a regression test that PolicyPal's mock can run. On submit, your tests must pass on a correct PolicyPal and catch at least four of its five planted defects.

Your notes are compared with a model answer, not graded. Before a real model is switched on for this course, the console uses a stand-in and says so.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
The documents the model seesShow
leave-2025: Full-time employees get 24 days of paid annual leave a year. Up to 10 unused days carry over to the next year. Leave must be approved by your manager in the HR portal.
leave-2023 (replaced): Full-time employees get 18 days of paid annual leave a year. Unused leave does not carry over. This policy was replaced on 1 April 2025.
wfh-2025: Employees may work from home up to 2 days a week with their manager's approval. New joiners work from the office for their first 30 days.
travel-2025: Book business travel through the travel portal at least 7 days ahead. Flights under 6 hours are booked in economy class.
expenses-2025: Submit expense claims within 30 days of the expense, with receipts. Meals on business travel are reimbursed up to Rs 1,500 a day.
notice-2025: The notice period is 60 days for permanent employees and 30 days during probation.
probation-2025: Probation lasts 6 months from the joining date. Your manager confirms you in the role at the end of it.
parental-2025: Employees get 26 weeks of paid maternity leave and 2 weeks of paid paternity leave.
holidays-2026: There are 12 fixed public holidays in 2026, plus 2 restricted holidays you choose from the HR portal list.
laptop-2025: Every employee gets a company laptop. Report a lost or stolen laptop to IT within 24 hours on extension 4040.
health-2025: Health insurance is a Rs 5 lakh family floater covering you, your spouse, two children and your parents.
referral-2025: The referral bonus is Rs 50,000, paid after the person you referred completes 90 days.
1

Real model with PolicyPal's documents

Real model36 of 36 messages left in this lesson

No messages yet.

Rounds sent: 0 of 1
2

Write it up

Categories: stale_source, wrong_citation, unsupported_claim, invented_answer, over_refusal, access_leak, other, held_up.

How this is checked: Each finding must cite a trace ID from your own console log. When you submit, the server checks that it is valid JSON and follows the rules in the task. Your result names any rule it breaks. Compare it with the model answer after you submit.

3

Your code

against PolicyPal's mock

How this is checked: When you submit, your tests run against 6 versions of PolicyPal: 1 correct and 5 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Find real LLM failures and turn them into regression tests a mock can run

AI Testing for QA EngineersAdvancedTesting RAG and hallucination

How it works

Planted defects teach the shape of failures; real models fail in ways nobody planted. Sending a fixed set of questions to a real model with PolicyPal's documents as context, and reading every reply against those documents, is how a team learns what its own model actually gets wrong.

Real failures are often subtle. A model may agree with a wrong number the user suggested, answer confidently about a topic no document covers, or cite a document that only half supports the sentence. Each one needs a trace ID from your own log, so anyone can reopen it.

A finding is only finished when it becomes a regression test. Because the real model is expensive and changes, the regression tests run against PolicyPal's mock, phrased as checks the mock can answer.

About the assignment

Send the lesson's questions to the real model, record three findings as JSON with trace IDs and categories, and write regression tests for their kinds of defect.

What you will do

  1. Send the whole question set with one click, then read every reply with the documents open.
  2. For each failure, copy the trace ID from your log and choose the closest category.
  3. Write each regression test so it fails on the mock build with that kind of defect.

Common mistakes

  • Invented trace IDs

    Evidence must point at a call in your own log. A trace ID from elsewhere counts for nothing.

  • A test tied to the real model's wording

    The mock words things differently. Test the kind of defect, not the exact sentence you saw.

  • Inventing a failure to fill the list

    If the model answered a tricky question correctly, record it as held_up. A false finding wastes the time of whoever reopens it.

In a real test suite

Teams run a hunt like this before each model change and after each major prompt change. The findings feed the guard set, which is how an eval suite grows from real failures instead of guesses.

Keeping the categories small and fixed makes findings from different people comparable. A free-text category list grows until nobody can count anything.

Related lessons

Next lessonGrade the outcome, not the claimAll lessons in the courseAI Testing for QA Engineers