Skip to content
CodeKruPro

Heal or reject

32/99

Module · Reviewing AI-written tests

intermediate

about 35 min

What this lesson teaches

The idea

Self-healing tools fix failing tests automatically. Some fixes are right; others make a test pass by making it stop testing.

Why a tester cares

One question sorts them: did the app change on purpose, or did the test stop checking? Expected values follow the spec, never the screen.

Words you will meet

Self-healing:
a tool that rewrites a failing test so it passes again.
Locator update:
pointing a test at an element whose name or position changed.

What you will do

  1. 1Read the eight diffs
  2. 2Accept or reject
  3. 3Repair two tests
  4. 4Submit
Jump to practice

Self-healing tools and test agents fix failing tests automatically. Some fixes are exactly right: a button really was renamed, so the locator should follow. Others make a failing test pass by making it stop testing: changing the expected total to whatever the app now shows, skipping the test, or catching the assertion error.

The question for every healing diff is the one a reviewer asks of any change: did the app change on purpose, or did the test stop checking? A locator follows intended UI changes. An expected value follows the spec, never the screen.

Your task

  1. Accept or reject each of the eight healing diffs.
  2. Repair the two healed tests in the editor so they check the spec again. No skips and no fixme.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
ShopLab specShow
ShopLab pricing and checkout rules

Products: Coffee mug (MUG) Rs 299 · Notebook (NOTE) Rs 149 · Wireless headphones (PHONES) Rs 1,999 · Cotton T-shirt (TEE) Rs 499
Each cart line has a quantity from 1 to 10.

Coupons (one per order; applying a new coupon replaces the old one)
  SAVE10   10% off the subtotal, rounded down to the rupee. Needs a subtotal of Rs 500 or more.
           Below that the message is "SAVE10 needs a subtotal of Rs 500 or more." and the discount stays Rs 0.
  FLAT100  Rs 100 off. Needs a subtotal of Rs 999 or more.
  OLD20    Expired. Message: "This coupon has expired."
  Anything else: "Coupon not found."

Shipping: Rs 49 when the subtotal after discount is under Rs 500; free from Rs 500.
Total = subtotal - discount + shipping.

Confirmation page: heading "Order confirmed", the order ID (SL-12345), one line per item such as
"Notebook × 2", and "Total: Rs 1,234". Item lines and the total load a moment after the page.

For tests
  The app's address is in the CK_APP_URL environment variable. Pages: /shop/, /shop/cart, /shop/help.
  POST {CK_APP_URL}/api/cart with {"sku": "MUG", "qty": 2} fills the cart. page.request shares the browser's
  cookies, so a test can set up its cart through the API and then check the page.
  Test IDs (data-testid): message, subtotal, discount, shipping, total, order-id, order-line, order-total.

Eight healing diffs

Each card shows why the agent changed the test and the diff it made.

D1After the button was renamed in a release (changelog: "Place order" is now "Complete purchase"):
  1. 1. Diff- page.get_by_role("button", name="Place order") + page.get_by_role("button", name="Complete purchase")
D2The confirmation test failed after a deploy:
  1. 1. Diff- expect(page.get_by_test_id("order-total")).to_have_text("Total: Rs 539") + expect(page.get_by_test_id("order-total")).to_have_text("Total: Rs 588")
D3A coupon test failed three times in a row:
  1. 1. Diff- def test_expired_coupon(page): + @pytest.mark.skip(reason="flaky, healed") + def test_expired_coupon(page):
D4The order total sometimes was not there yet:
  1. 1. Diff+ page.wait_for_timeout(5000) total = page.get_by_test_id("order-total").inner_text()
D5A selector broke after a layout change:
  1. 1. Diff- page.locator("dl > dd:nth-of-type(4)") + page.get_by_test_id("total")
D6The heading text check failed after a copy change was reverted:
  1. 1. Diff- expect(page.get_by_role("heading", name="Order confirmed")).to_have_text("Order confirmed") + expect(page.locator("h1")).to_be_visible()
D7A shipping assertion failed on one build:
  1. 1. Diff- expect(page.get_by_test_id("shipping")).to_have_text("Free") + try: + expect(page.get_by_test_id("shipping")).to_have_text("Free") + except AssertionError: + print("warning: shipping text differs")
D8After the coupon field's label changed (spec change PR-412 approved by product):
  1. 1. Diff- page.get_by_label("Coupon code") + page.get_by_label("Promo code")
1

Label each one

Accept or reject each diff

0 of 8 labelled

  1. 1D1: After the button was renamed in a release (changelog: "Place order" is now "Complete purchase"):
  2. 2D2: The confirmation test failed after a deploy:
  3. 3D3: A coupon test failed three times in a row:
  4. 4D4: The order total sometimes was not there yet:
  5. 5D5: A selector broke after a layout change:
  6. 6D6: The heading text check failed after a copy change was reverted:
  7. 7D7: A shipping assertion failed on one build:
  8. 8D8: After the coupon field's label changed (spec change PR-412 approved by product):

How this is checked: Each row has a right label. At least 6 of 8 must match. For the full score, a harder check: a few key rows must all be right.

2

Your code

Write it in
Both are graded the same way.
against today's ShopLab

How this is checked: When you submit, your tests run against 3 versions of the ShopLab store: 1 correct and 2 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Review self healing test changes: accept the locator fix, reject the weakened assertion

AI Testing for QA EngineersIntermediateReviewing AI-written tests

How it works

Self-healing tools and test agents repair failing tests automatically. Some repairs are right: a button really was renamed, and the locator should follow. Others make a failing test pass by making it stop testing, by changing the expected value to whatever the app shows, skipping the test, or swallowing the assertion error.

One question sorts them: did the app change on purpose, or did the test stop checking? Locators may follow intended UI changes. Expected values follow the spec and never the screen.

About the assignment

Accept or reject eight healing diffs, then repair two healed tests by hand so they check what they were meant to.

What you will do

  1. Read each diff and name what it changed: locator, expected value, skip, or error handling.
  2. Accept locator updates that follow a deliberate UI change.
  3. Reject anything that weakens what the test asserts, and repair those tests from the spec.

Common mistakes

  • Accepting a new expected total

    If the healer changed a total to match the app, the bug it should have caught is now hidden.

  • Rejecting every change

    A renamed button is a legitimate change. Blocking every heal makes the tool useless.

In a real test suite

Teams that use healing tools usually let locator heals through automatically and route every change to an assertion or a skip to a human reviewer.

A useful review habit is to read the healed test as if a colleague had written it from scratch. If you would not approve that test in a normal pull request, you should not approve it because a tool produced it.

Related lessons

Next lessonWhat the agent seesAll lessons in the courseAI Testing for QA Engineers