A generated test suite that passes has told you nothing. The honest measure is what it catches when you run it on builds that each contain one known bug.
Why a tester cares
Every bug that gets through shows either a missing test or a weak check, and each one tells you exactly what to add.
"The tests pass" says nothing about a generated suite. What it is worth is what it catches. The quickest honest measure: run it against builds with known bugs and count the escapes.
This lesson's Run uses five ShopLab builds: one correct, and four with a coupon bug each, named by the bug. On a buggy build, a green run means the bug escaped.
Your task
Ask the assistant for five coupon tests (the suggestion works), put them in the editor and Run.
Record which bugs were caught and which escaped.
Fix the suite: add or change tests until every bug is caught and the correct build still passes. Keep your prompt in the box; a reviewer will want to see it.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
ShopLab pricing and checkout rules
Products: Coffee mug (MUG) Rs 299 · Notebook (NOTE) Rs 149 · Wireless headphones (PHONES) Rs 1,999 · Cotton T-shirt (TEE) Rs 499
Each cart line has a quantity from 1 to 10.
Coupons (one per order; applying a new coupon replaces the old one)
SAVE10 10% off the subtotal, rounded down to the rupee. Needs a subtotal of Rs 500 or more.
Below that the message is "SAVE10 needs a subtotal of Rs 500 or more." and the discount stays Rs 0.
FLAT100 Rs 100 off. Needs a subtotal of Rs 999 or more.
OLD20 Expired. Message: "This coupon has expired."
Anything else: "Coupon not found."
Shipping: Rs 49 when the subtotal after discount is under Rs 500; free from Rs 500.
Total = subtotal - discount + shipping.
Confirmation page: heading "Order confirmed", the order ID (SL-12345), one line per item such as
"Notebook × 2", and "Total: Rs 1,234". Item lines and the total load a moment after the page.
For tests
The app's address is in the CK_APP_URL environment variable. Pages: /shop/, /shop/cart, /shop/help.
POST {CK_APP_URL}/api/cart with {"sku": "MUG", "qty": 2} fills the cart. page.request shares the browser's
cookies, so a test can set up its cart through the API and then check the page.
Test IDs (data-testid): message, subtotal, discount, shipping, total, order-id, order-line, order-total.
1
Scripted assistant
No messages yet. Start with a suggested question below.
2
Your code
Write it inBoth are graded the same way.
1# Paste the suite the assistant writes for you here,then Run it.
2# Use "Put in the editor" under the assistant's reply,or copy it by hand.
3
on the correct build and four buggy builds
How this is checked: When you submit, your tests run against 6 versions of the ShopLab store: 2 correct and 4 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.
3
Label each one
With the assistant's five tests, before your fixes
0 of 4 labelled
1Bug A: SAVE10 works below Rs 500
2Bug B: the expired OLD20 still works
3Bug C: a second coupon adds to the first
4Bug D: SAVE10 rounds the discount up
How this is checked: Each row has a right label. All 4 must match.
4
Write it up
How this is checked: It only needs an answer. This is a harder check: it earns the second 50 points once the core checks pass.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Measure what AI generated tests catch: run them against builds with known bugs
AI Testing for QA Engineers·Intermediate·Working with an AI assistant as a tester
How it works
A generated suite that passes has told you nothing about its value. The honest measure is what it catches: run it against builds that each contain one known bug, and count the bugs that escape. A green run on a buggy build is an escape.
Escapes usually come from two places. A missing test means nothing exercises the behaviour, such as applying two coupons in a row. A weak assertion means a test exercises it but checks too little, such as "the discount is not zero" when the spec fixes the exact amount.
Fixing the suite is then specific work: one new test per missing behaviour, and one stronger assertion per weak check, until every bug fails and the correct build still passes.
About the assignment
Generate five coupon tests with the scripted assistant, run them on one correct and four buggy ShopLab builds, record what escaped, and improve the suite until nothing does.
What you will do
1Put the assistant's suite in the editor and Run it on all five builds.
2Mark each bug caught or escaped from its build's result.
3For each escape, decide whether a test is missing or an assertion is weak, then fix that.
4Keep the prompt you used, so a reviewer can reproduce the suite.
Common mistakes
Reading only the correct build
Passing on the correct build is the minimum. The buggy builds are where the suite earns its keep.
Changing expected values to match a bug
Expected values come from the spec. If a test fails on the correct build, the test is wrong, not the spec.
In a real test suite
Running generated tests against known-bad builds is a cheap form of mutation testing, and it turns a code review argument into a count everyone can see.
