Skip to content
CodeKruPro

Give the assistant what a tester knows

24/99

Module · Working with an AI assistant as a tester

intermediate

about 40 min

What this lesson teaches

The idea

AI assistants write tests quickly, and most of what they write passes. Whether those tests catch anything depends on what you tell the assistant.

Why a tester cares

Three things only a tester tends to supply: the rule from the spec, the exact expected value, and the cases that must be rejected.

Words you will meet

Prompt:
the instructions you give an AI assistant.
Oracle:
the exact value a test must check, worked out from the spec.
Negative case:
an input that must be rejected, such as a coupon below its minimum order.

What you will do

  1. 1Try the vague prompt
  2. 2Write a better one
  3. 3Run the new suite
  4. 4Submit
Jump to practice

AI assistants write tests quickly, and most of what they write passes. That is the problem: a suite that passes on a buggy app is worse than no suite, because it makes people stop looking. What separates a useful generated suite from a decorative one is usually not the assistant. It is what the tester put in the prompt.

Three things only a tester tends to say:

  • The rule: what the feature must do, from the spec, including limits such as a minimum order.
  • The oracle: the exact result a test must assert, worked out from the rule, not "check it works".
  • The negative cases: what must be rejected or must fail.

The assistant in this lesson is scripted: it returns recorded suites chosen by which of those three your prompt contains. A real assistant varies more, but responds to the same things.

Your task

  1. Send the vague prompt below to the assistant and Run what comes back. Notice how green it is.
  2. Rewrite the prompt with the rule, the oracle and the negative cases from the ShopLab spec. Put the suite it returns in the editor and Run it.
  3. Paste your final prompt in the box and Submit. Your prompt is checked, and your suite runs against hidden ShopLab builds.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
ShopLab specShow
ShopLab pricing and checkout rules

Products: Coffee mug (MUG) Rs 299 · Notebook (NOTE) Rs 149 · Wireless headphones (PHONES) Rs 1,999 · Cotton T-shirt (TEE) Rs 499
Each cart line has a quantity from 1 to 10.

Coupons (one per order; applying a new coupon replaces the old one)
  SAVE10   10% off the subtotal, rounded down to the rupee. Needs a subtotal of Rs 500 or more.
           Below that the message is "SAVE10 needs a subtotal of Rs 500 or more." and the discount stays Rs 0.
  FLAT100  Rs 100 off. Needs a subtotal of Rs 999 or more.
  OLD20    Expired. Message: "This coupon has expired."
  Anything else: "Coupon not found."

Shipping: Rs 49 when the subtotal after discount is under Rs 500; free from Rs 500.
Total = subtotal - discount + shipping.

Confirmation page: heading "Order confirmed", the order ID (SL-12345), one line per item such as
"Notebook × 2", and "Total: Rs 1,234". Item lines and the total load a moment after the page.

For tests
  The app's address is in the CK_APP_URL environment variable. Pages: /shop/, /shop/cart, /shop/help.
  POST {CK_APP_URL}/api/cart with {"sku": "MUG", "qty": 2} fills the cart. page.request shares the browser's
  cookies, so a test can set up its cart through the API and then check the page.
  Test IDs (data-testid): message, subtotal, discount, shipping, total, order-id, order-line, order-total.
1

Scripted assistant

No messages yet. Start with a suggested question below.

2

Write it up

The one whose suite you are submitting.

How this is checked: When you submit, the server checks it for what the task asks: your prompt gives the rule, the exact result and a negative case.

3

Your code

Write it in
Both are graded the same way.
against ShopLab

How this is checked: When you submit, your tests run against 5 versions of the ShopLab store: 2 correct and 3 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Prompt an AI assistant to write better tests: the rule, the oracle and the negative cases

AI Testing for QA EngineersIntermediateWorking with an AI assistant as a tester

How it works

Assistants write tests quickly, and most of what they write passes. A green suite on a buggy app is worse than none, because it makes people stop looking. The difference between a useful generated suite and a decorative one is usually the prompt.

Three things only a tester tends to supply: the rule from the spec, including limits; the oracle, meaning the exact value a test must assert; and the negative cases that must be rejected. The scripted assistant in this lesson responds to those three, and real assistants respond to the same signals.

About the assignment

Send a vague prompt, run the suite it produces, then write a prompt that supplies the rule, the oracle and the negative cases, and run the new suite against the builds.

What you will do

  1. Run the suite from the vague prompt and note how green it is.
  2. Add the coupon rule and its minimum from the spec.
  3. Give an exact expected total and at least one case that must fail.

Common mistakes

  • Asking for "more tests"

    Volume does not help. The missing ingredient is what each test should assert.

  • An oracle copied from the screen

    Work the expected value out from the spec, or a bug becomes the expected result.

In a real test suite

Teams that get value from assistants keep a prompt library and an instruction file per repository, and they always run generated tests against a known-broken build before merging.

Related lessons

Next lessonInstruction files for test generationAll lessons in the courseAI Testing for QA Engineers