Skip to content
CodeKruPro

When the set stops representing traffic

40/99

Module · Datasets

intermediate

about 40 min

What this lesson teaches

The idea

A test set is a snapshot of what worried someone when it was written. Real traffic moves on: new customers, new languages, new questions.

Why a tester cares

When the set stops looking like traffic, a green run means less each month. You find the gap by comparing the set's mix with traffic, group by group.

Words you will meet

Drift:
the slow change of real traffic away from what the test set covers.
Segment:
one group of questions, such as Hindi questions or gift-card questions.
Owner:
the person who decides the right answer for a case.

What you will do

  1. 1Compare set and traffic
  2. 2Measure the gap
  3. 3Add cases
  4. 4Version the set
Jump to practice

A golden set is a snapshot of what you worried about when you wrote it. Traffic moves on: new customers, new languages, new questions. When the set no longer looks like traffic, a green run tells you less every month. This is dataset drift, and it is found by comparing the set's mix with real traffic, not by reading failures.

Every case also needs an owner: the person who can say what the right answer is. Policy facts belong to the support lead; wording in a language you do not speak belongs to someone who does.

Your task

  1. Compare the set with last month's traffic and measure the gap for the segment that drifted.
  2. Add at least four cases for that segment. Keep every case's language and owner fields filled.
  3. Write a short version note: what changed, why, and who owns what.

On submit the updated set runs against a correct RefundBot and against a build that answers that segment wrongly.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
Last month's trafficShow
Last month's RefundBot traffic (10,000 chats), by intent and language

intent             english   hinglish   hindi
return window        1,620      1,240      160
refund time          1,410        620       70
damaged item           980        410       60
gift card              560        140       10
pickup fee             480        220       20
eligibility          1,120        560       80
other                  220         20        0

Share of chats in Hinglish: 3,210 of 10,000 = 32.1%
Six months ago Hinglish was 4% of chats.
1

Question

Which segment drifted most?

How this is checked: One answer is right.

2

Your answer

For example 12.5 for 12.5 points.

How this is checked: Your number must match the right value; small rounding differences are fine.

3

Write it up

How this is checked: When you submit, the server checks that it is valid JSONL and follows the rules in the task. Your result names any rule it breaks. This is a harder check: it earns the second 50 points once the core checks pass. When you submit, it is saved as golden.jsonl and used by the run.

4

Write it up

How this is checked: When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks. This is a harder check: it earns the second 50 points once the core checks pass.

5

Your code

Provided by the lesson. You can read it but not change it.

your set against RefundBot

How this is checked: When you submit, your script runs on 2 sets of data, 2 of them hidden, and the server compares what it prints with the right values.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Dataset drift in a golden test set: comparing the set with real traffic

AI Testing for QA EngineersIntermediateDatasets

How it works

A golden set is a snapshot of what worried someone when it was written. Traffic moves on: new customers, new languages, new questions. When the set no longer looks like traffic, a green run tells you less each month.

Drift is found by comparing the set's mix with real traffic, segment by segment, not by reading failures. Every case also needs an owner: the person who can say what the right answer is, such as the support lead for policy facts or a fluent speaker for another language.

About the assignment

Compare the set with last month's traffic, measure the gap in the drifted segment, add cases for it with owners, and version the set.

What you will do

  1. Compute each segment's share in the set and in traffic.
  2. Find the segment with the biggest gap.
  3. Add cases for it, give each an owner, and bump the set's version.

Common mistakes

  • Owners who cannot judge

    An owner must be able to say what correct is for that case.

  • Editing in place

    Without a new version, results before and after the change cannot be compared.

In a real test suite

Teams schedule drift checks, often monthly, and keep the set's version in every run manifest so scores from different versions are never compared directly.

Adding cases is only half the fix. Removing or down-weighting segments that traffic no longer contains keeps run time and cost in check, and keeps the overall pass rate from being dominated by questions customers have stopped asking.

Related lessons

Next lessonA pass rate is a rangeAll lessons in the courseAI Testing for QA Engineers