Skip to content
CodeKruPro

Why a green AI-written suite can miss everything

28/99

Module · Reviewing AI-written tests

intermediate

about 30 min

What this lesson teaches

The idea

High coverage feels safe, but it only records which lines ran, not whether any test would notice them doing the wrong thing.

Why a tester cares

AI-written suites often check only that things exist, which always passes. Running them on buggy builds shows how much they really miss.

Words you will meet

Code coverage:
the share of code lines that ran during the tests.
Escape:
a bug the suite passes without noticing.
Existence check:
a test that only checks an element is on the page.

What you will do

  1. 1Run the suite
  2. 2Mark caught or escaped
  3. 3Name the weakness
  4. 4Submit
Jump to practice

A suite that passes and covers 91% of the code feels safe. Coverage only tells you which lines ran, not whether any test would notice if they did the wrong thing. AI-generated suites are especially prone to this: they visit every page and assert that things exist.

The suite in the editor was written from a one-line prompt. Run it: the first build is correct, and the next five each have one bug, named by what is wrong.

Your task

  1. Run the suite and read each build's result. A green run on a buggy build means the bug escaped.
  2. Mark each bug caught or escaped, and count the escapes.
  3. Name the weakness that let them through.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
ShopLab specShow
ShopLab pricing and checkout rules

Products: Coffee mug (MUG) Rs 299 · Notebook (NOTE) Rs 149 · Wireless headphones (PHONES) Rs 1,999 · Cotton T-shirt (TEE) Rs 499
Each cart line has a quantity from 1 to 10.

Coupons (one per order; applying a new coupon replaces the old one)
  SAVE10   10% off the subtotal, rounded down to the rupee. Needs a subtotal of Rs 500 or more.
           Below that the message is "SAVE10 needs a subtotal of Rs 500 or more." and the discount stays Rs 0.
  FLAT100  Rs 100 off. Needs a subtotal of Rs 999 or more.
  OLD20    Expired. Message: "This coupon has expired."
  Anything else: "Coupon not found."

Shipping: Rs 49 when the subtotal after discount is under Rs 500; free from Rs 500.
Total = subtotal - discount + shipping.

Confirmation page: heading "Order confirmed", the order ID (SL-12345), one line per item such as
"Notebook × 2", and "Total: Rs 1,234". Item lines and the total load a moment after the page.

For tests
  The app's address is in the CK_APP_URL environment variable. Pages: /shop/, /shop/cart, /shop/help.
  POST {CK_APP_URL}/api/cart with {"sku": "MUG", "qty": 2} fills the cart. page.request shares the browser's
  cookies, so a test can set up its cart through the API and then check the page.
  Test IDs (data-testid): message, subtotal, discount, shipping, total, order-id, order-line, order-total.
Coverage report from the team's CIShow
cart.js          91% of lines executed
orders.js        88% of lines executed
coupons.js       94% of lines executed

All 6 tests passed on main.
1

Your code

Provided by the lesson. You can read it but not change it.

on the correct build and five buggy builds
2

Label each one

Did the suite catch each bug?

0 of 5 labelled

  1. 1Bug 1: confirmation total adds shipping when it is free
  2. 2Bug 2: confirmation shows quantity 1 for every line
  3. 3Bug 3: confirmation shows no items and no total
  4. 4Bug 4: SAVE10 works below Rs 500
  5. 5Bug 5: SAVE10 rounds the discount up

How this is checked: Each row has a right label. All 5 must match.

3

Your answer

How this is checked: Your number must match the right value.

4

Question

What let the bugs through?

How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the core checks pass.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Code coverage vs bug detection in AI generated tests: why a green suite can miss everything

AI Testing for QA EngineersIntermediateReviewing AI-written tests

How it works

High coverage feels safe, but coverage only records which lines ran, not whether any test would notice those lines doing the wrong thing. A suite that visits every page and asserts that elements exist can reach a high coverage figure while catching almost nothing.

AI-generated suites lean this way because existence checks always pass. The quickest way to see it is to run the suite on builds that each carry one named bug and read the results: every green run on a buggy build is a bug the suite would have let through.

The pattern behind the escapes is usually the same weakness repeated across tests, which makes it easy to name and to fix in the next lesson.

About the assignment

Run a suite written from a one-line prompt against a correct build and five buggy ones, mark each bug caught or escaped, count the escapes, and name the weakness.

What you will do

  1. Run the suite and open each build's result.
  2. Treat a passing run on a buggy build as an escape.
  3. Look at what the tests assert, and name the habit that let the bugs through.

Common mistakes

  • Equating coverage with safety

    A covered line with no meaningful assertion is still untested.

  • Blaming the bugs

    The bugs are planted on purpose. The finding is about the suite.

In a real test suite

Some teams report escapes next to coverage in pull requests that add generated tests. The pair tells reviewers far more than coverage alone.

Related lessons

Next lessonFix URL-only and visible-only assertionsAll lessons in the courseAI Testing for QA Engineers