Release evidence rarely agrees. Evals are green, the A/B test is positive, and the support lead has found something that worries them. A tester's job is not to pick the most comfortable number, but to apply the rules the team agreed and say plainly what is still unknown.
Two things make that easier. First, rules agreed before the release: when the evidence arrives, you apply them rather than argue. Second, the question "what did each source actually measure?". An A/B test with 3% damaged-item traffic cannot speak for damaged items.
Your task
Read the evidence and decide each case by the team's rules.
Write a one-page memo for this release. Compare it with the model memo after you submit.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
Release: RefundBot prompt v6 with the new damaged-item flow
Evals: golden set 96.2% (baseline 95.4%), 120 cases, 3 runs each; the 95% intervals overlap.
Safety suite: 100% on 65 hard-gate cases.
A/B test: 7 days, 18,000 conversations; self-serve resolution 61.8% vs 60.6% (p = 0.04). Damaged-item
conversations were 3% of traffic in the test.
Support lead: reviewed 50 damaged-item conversations from the test arm by hand and found 3 where the bot
promised a refund amount higher than the order value. The golden set has 4 damaged-item cases.
Release rules the team agreed:
R1 A confirmed failure in a hard-gate category (money, safety, personal data) holds the release.
R2 Positive evidence with a segment that was not measured ships behind a flag for that segment.
R3 Green evals and a positive or neutral A/B test ship.
R4 Evals that are worse beyond their interval hold, whatever the A/B test says.
1
Label each one
Ship, ship behind a flag, or hold?
0 of 6 labelled
1This release, as described above.
2Same, but the support lead's 3 cases turn out to be correct refunds the reviewer misread.
3Same as d2, but the new flow also adds Hinglish replies, which no eval or test covered.
4Evals drop from 95.4% to 91.0%, outside the interval, while the A/B test is positive.
6Evals up, A/B positive, but one safety case now leaks a phone number.
How this is checked: Each row has a right label. All 6 must match.
2
Write it up
How this is checked: Not marked for content: it needs at least 120 words. When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks. This is a harder check: it earns the second 50 points once the core checks pass. Compare it with the model answer after you submit.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Ship or hold decision for an AI feature release with conflicting evidence, and a memo with uncertainty
AI Testing for QA Engineers·Advanced·Release, production and operations
How it works
Release evidence rarely agrees. Evals are green, the A/B test is positive, and the support lead has found something worrying. A tester's job is not to pick the most comfortable number but to apply the rules the team agreed and say plainly what is still unknown.
Two habits make that easier. Rules agreed before the release let you apply them rather than argue. And asking what each source actually measured shows its limits: an A/B test with a tiny share of damaged-item traffic cannot speak for damaged items.
About the assignment
Decide six release situations by the team's rules, then write a one-page memo for the main release with a recommendation, evidence, uncertainty, what would change your mind, and an owner.
What you will do
1Apply the rules in order for each situation.
2Check what population each piece of evidence came from.
3Write the memo so someone outside the team could act on it.
Common mistakes
Letting a positive A/B test override a hard-gate failure
A confirmed money failure holds the release under the first rule.
A memo without uncertainty
Saying what you do not know is what makes the recommendation trustworthy.
In a real test suite
Memos like this become the record of why a release was held or shipped. Teams that keep them can look back after an incident and see whether the rules or their application need to change.
The section on what would change your mind turns a refusal into a plan, which keeps product and QA working together instead of against each other.
