When a test fails, the first suspect should sometimes be the test. Gold sets go stale: a policy changes, the bot is updated, and nobody updates the expected answers. Every stale label turns a correct reply into a "failure", or a wrong reply into a pass.
The 40-row set below was written two policy versions ago. About 15% of its expected answers are out of date.
Your task
Fix every stale expected value so it matches the current policy, using the exact values listed. Do not change rows that are already right, and keep every row.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
Return window: 30 days
Refund time: 5 to 7 business days
Report damage within: 48 hours
Return pickup: Rs 99
Pickup for damaged items: free
Gift cards: not refundable
Use exactly these values in "expected".
1
Write it up
How this is checked: When you submit, each row's expected is compared with the right value.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Fix stale labels in an eval dataset when the policy has changed
AI Testing for QA Engineers·Intermediate·Datasets
How it works
When a test fails, sometimes the test is the suspect. Gold sets go stale: a policy changes, the bot is updated, and the expected answers stay as they were. Each stale label turns a correct reply into a failure, or a wrong reply into a pass.
Auditing means checking every expected value against the current source of truth and changing only those that are out of date. Changing correct rows, or dropping rows that look awkward, damages the set in a different way.
About the assignment
Fix every stale expected value in a 40-row set so it matches the current policy, leaving correct rows untouched and keeping every row.
What you will do
1Read the current policy and list the values that changed.
2Check each row's expected value against that list.
3Edit only the stale values, and keep every row and field.
Common mistakes
Rewriting correct rows
A row that already matches the policy must stay exactly as it is.
Deleting rows
A missing row hides the case it tested. Fix it instead.
In a real test suite
Teams tie gold sets to the documents they depend on, so a policy change triggers a review of the affected rows rather than waiting for tests to fail.
After fixing the labels, rerun the set on the current bot. Several failures that people had learned to ignore often disappear, and the ones that remain are now worth investigating, because the set finally agrees with the policy it is meant to protect.
