Skip to content
CodeKruPro

Capstone 1: PolicyPal evaluation report

96/99

Module · Career, capstones and the Cold Product Test

advanced

about 180 min

What this lesson teaches

The idea

This capstone evaluates PolicyPal end to end and produces what a release owner reads: a failure taxonomy, a case set, a test suite, defect reports and a short report.

Why a tester cares

Your copy of PolicyPal has its own set of defects, so your findings are your own. It brings together everything from the RAG lessons.

Words you will meet

Capstone:
a final project that combines a whole part of the course.
Taxonomy:
named groups of ways the product can fail.
Stale trap:
a question whose answer changed, to catch a bot using an old document.

What you will do

  1. 1Map the failures
  2. 2Write 40 cases
  3. 3Test every layer
  4. 4Hunt and report
  5. 5Write the report
Jump to practice

This capstone puts Modules 4 and 11 together on one product. You evaluate PolicyPal, Acme's HR policy bot, the way you would on a real team, and produce the five things a release owner reads.

  1. Taxonomy: the ways PolicyPal can fail, each with its layer: retrieval, generation, access or behaviour.
  2. A 40-case set covering answerable, unanswerable, restricted and stale-trap questions, each with a pass criterion.
  3. A test suite that reads your set. On submit it runs against two correct builds and seven faulty ones you cannot see.
  4. Defect reports from your own PolicyPal build. It has four of the seven defects, chosen for you, so your list will differ from other learners'. Cite a trace from your console for each, and report nothing that works correctly.
  5. A report with scope, method, results, defects, and risks and limits. Compare yours with the model report after you submit.

You are a Sales employee throughout.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
PolicyPal's documents (for everyone)Show
leave-2025: Full-time employees get 24 days of paid annual leave a year. Up to 10 unused days carry over to the next year. Leave must be approved by your manager in the HR portal.
leave-2023 (replaced): Full-time employees get 18 days of paid annual leave a year. Unused leave does not carry over. This policy was replaced on 1 April 2025.
wfh-2025: Employees may work from home up to 2 days a week with their manager's approval. New joiners work from the office for their first 30 days.
travel-2025: Book business travel through the travel portal at least 7 days ahead. Flights under 6 hours are booked in economy class.
expenses-2025: Submit expense claims within 30 days of the expense, with receipts. Meals on business travel are reimbursed up to Rs 1,500 a day.
notice-2025: The notice period is 60 days for permanent employees and 30 days during probation.
probation-2025: Probation lasts 6 months from the joining date. Your manager confirms you in the role at the end of it.
parental-2025: Employees get 26 weeks of paid maternity leave and 2 weeks of paid paternity leave.
holidays-2026: There are 12 fixed public holidays in 2026, plus 2 restricted holidays you choose from the HR portal list.
laptop-2025: Every employee gets a company laptop. Report a lost or stolen laptop to IT within 24 hours on extension 4040.
health-2025: Health insurance is a Rs 5 lakh family floater covering you, your spouse, two children and your parents.
referral-2025: The referral bonus is Rs 50,000, paid after the person you referred completes 90 days.

Restricted: eng-bands-2025 (Engineering managers only).
PolicyPal APIShow
PolicyPal's API, at the address in CK_APP_URL:
  POST /policy/ask {"question": "...", "department": "sales"} -> {"answer", "citations": [ids], "retrieved", "trace_id"}
  GET  /policy/docs/<id>?department=sales -> the document, 404 if it does not exist, 403 if restricted
  GET  /policy/search?q=...&k=3 -> ranked documents
1

Write it up

How this is checked: When you submit, the server checks that it is valid JSON and follows the rules in the task. Your result names any rule it breaks. Compare it with the model answer after you submit.

2

Write it up

Categories: answerable (with expect_doc), unanswerable, restricted, stale_trap. must_match is a regex the right answer contains.

How this is checked: When you submit, the server checks that it is valid JSONL and follows the rules in the task. Your result names any rule it breaks. When you submit, it is saved as cases.jsonl and used by the run.

3

Your code

your set against PolicyPal

How this is checked: When you submit, your tests run against 9 versions of PolicyPal: 2 correct and 7 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

4

Your PolicyPal build (you are in Sales)

No messages yet. Start with a suggested question below.

5

Write it up

How this is checked: When you submit, each report is matched with the faults planted in your copy of the product, and each must cite a trace ID from your own log. Reports that match nothing count against you.

6

Write it up

How this is checked: When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks. This is a harder check: it earns the second 50 points once the core checks pass. Compare it with the model answer after you submit.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

RAG chatbot evaluation report capstone: taxonomy, cases, test suite, defects and report

AI Testing for QA EngineersAdvancedCareer, capstones and the Cold Product Test

How it works

This capstone evaluates PolicyPal end to end and produces the five things a release owner reads. A taxonomy names the ways it can fail, each with its layer. A case set covers answerable, unanswerable, restricted and stale-trap questions with pass criteria. A test suite reads the set and runs against correct and faulty builds you cannot see.

Defect reports come from your own PolicyPal build, which has a set of defects chosen for you, so your list differs from other learners'. Each report cites a trace from your console and nothing that works is reported. A short report with scope, method, results, defects, and risks and limits ties it together.

About the assignment

Produce the taxonomy, the case set, the suite, the defect reports and the report for your PolicyPal build.

What you will do

  1. Explore your build and the documents before writing anything.
  2. Write cases whose patterns any correct wording passes, and run them on a correct build.
  3. Make sure every layer of your taxonomy has at least one check in the suite.
  4. Report only defects you reproduced, each with a trace ID.

Common mistakes

  • Copying another learner's defect list

    Builds differ. Reporting a defect your build does not have counts as a false report.

  • Checks tied to exact wording

    Correct builds phrase answers differently. Check facts, citations and support instead.

In a real test suite

A report in this shape is what a team expects after a pre-release evaluation. Put it, with its data and code, in your portfolio repository: it is the strongest single artefact the course produces.

Related lessons

Next lessonCapstone 2: OrderDesk release gateAll lessons in the courseAI Testing for QA Engineers