Skip to content
CodeKruPro

Build a 30-case golden set, and name its purpose

38/99

Module · Datasets

intermediate

about 60 min

What this lesson teaches

The idea

A golden set is a list of test questions with known good answers, run on every change. What it can claim depends on how the cases were chosen.

Why a tester cares

Thirty hand-picked cases tell you whether anything you care about broke. They do not tell you how often customers see failures.

Words you will meet

Golden set:
a fixed list of test cases with known good answers.
Guard set:
cases chosen on purpose to catch known failures, including hard ones.
Prevalence sample:
cases drawn at random from real traffic, so their pass rate estimates the real failure rate.

What you will do

  1. 1Choose the purpose
  2. 2Write 30 cases
  3. 3Run them
  4. 4Submit
Jump to practice

A golden set is a list of cases with known good answers that you run on every change. Before writing one, decide what it is for, because that decides what it can claim:

  • A guard set is hand-picked to catch known ways of failing, including hard and adversarial cases. Its pass rate tells you whether anything you care about broke. It does not tell you how often customers see a failure.
  • A prevalence sample is drawn at random from real traffic, so its pass rate estimates how often failures happen. It is mostly easy cases, so it catches little.

Thirty cases written by hand are a guard set. Cover the five kinds of question a support bot meets: answerable (the policy answers it), partial (part of it is answerable), unanswerable (about refunds, but not in the policy), out of scope, and adversarial (pushing the bot to say something false).

Your task

  1. Choose the set's purpose.
  2. Write 30 cases as JSONL. Each line: id, category, question, must_match (a regular expression that a correct reply contains) and, where useful, must_not_match.
  3. Run your set. It must pass on the correct RefundBot, so your patterns must accept every correct wording.

On submit your set runs against correct builds and against builds with planted defects. A good set passes the first and catches each of the second.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
RefundBot's policyShow
Most items can be returned within 30 days of delivery for a full refund.
Refunds reach the original payment method within 5 to 7 business days of Acme receiving the item.
Gift cards cannot be refunded.
Damaged items: report within 48 hours of delivery with a photo; pickup is free. Other return pickups cost Rs 99.
To reach a person: support@acme.example.
Questions outside returns, refunds and orders get a scope message ("…I can only answer questions about…").
1

Question

What is this set for?

How this is checked: One answer is right.

2

Question

Your set passes 29 of 30. What can you report?

How this is checked: One answer is right.

3

Write it up

One case per line. Categories: answerable, partial, unanswerable, out_of_scope, adversarial.

How this is checked: When you submit, the server checks that it is valid JSONL and follows the rules in the task. Your result names any rule it breaks. When you submit, it is saved as golden.jsonl and used by the run.

4

Your code

Provided by the lesson. You can read it but not change it.

your set against RefundBot

How this is checked: When you submit, your script runs on 5 sets of data, 3 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Build a golden dataset for chatbot testing, and know what its pass rate can claim

AI Testing for QA EngineersIntermediateDatasets

How it works

A golden set is a list of cases with known good answers that runs on every change. Its purpose decides what it can claim. A guard set is hand-picked to catch known ways of failing, including hard and adversarial cases: its pass rate tells you whether anything you care about broke. A prevalence sample is drawn at random from traffic: its pass rate estimates how often customers see failures.

Thirty hand-written cases are a guard set, so their pass rate is not an accuracy figure. Good guard sets cover every kind of question a support bot meets, avoid near-duplicates, and have patterns that any correct wording passes.

About the assignment

Choose the purpose of a 30-case set, write the cases across five categories as JSONL, and run them against RefundBot builds.

What you will do

  1. Decide the set's purpose and what its pass rate can claim.
  2. Write cases for answerable, partial, unanswerable, out-of-scope and adversarial questions.
  3. Run the set and loosen any pattern that fails a correct bot.

Common mistakes

  • Thirty versions of one question

    Near-duplicates cost money and add no coverage.

  • Calling the pass rate accuracy

    A hand-picked set says nothing about how often real customers see failures.

In a real test suite

Most teams keep both kinds of set: a guard set in CI, and a periodic random sample of production traffic labelled by people to estimate the real failure rate.

Write each case's pattern from the policy, not from a reply you saw. Then run the set on a correct bot: every failure there is a pattern that is too strict, and fixing it before the first real run saves the team from chasing false alarms later.

Related lessons

Next lessonAudit the goldAll lessons in the courseAI Testing for QA Engineers