Skip to content
CodeKruPro

Test a refund chatbot whose answers change every time

4/99

Module · Why exact match fails on AI

beginner

about 30 min

What this lesson teaches

The idea

A test that compares the whole reply with one sentence breaks as soon as the wording changes. A better test checks what must be true of every correct reply, such as "it says 30 days".

Why a tester cares

This is the first test you write that survives a chatbot rewording itself, and it is the pattern for most AI tests in this course.

Words you will meet

Assertion:
the line in a test that decides pass or fail.
Property:
something true of every correct reply, whatever the wording.
Build:
one version of the bot. Some builds are correct; some have a fault planted on purpose.
Regular expression (regex):
a pattern for finding text, such as "the number 30 followed by days".

What you will do

  1. 1Run the starter
  2. 2Check the fact
  3. 3Guard against leaks
  4. 4Submit
Jump to practice

A colleague wrote the test in the editor. It passed once, when they copied the reply they saw on screen. Since then it fails most days, and nobody trusts it.

When you submit, your test file runs against several builds of RefundBot that you cannot see. Some are correct and word their answers differently. Some have a fault planted in them. A good test passes on every correct build and fails on every faulty one.

Your task

  1. Click Run to see the starter test fail against the practice bot, and read why.
  2. Change the test so it checks the fact that matters (the 30-day window) instead of one exact sentence.
  3. Add a second test for something a support reply must never contain: personal data about another customer, such as an email address or a phone number.

Python regular expressions (re), patterns for finding text, are enough for this lesson. You do not need any other library.

Acme Store refund policy

  • Most items can be returned within 30 days of delivery for a full refund.
  • Refunds reach the original payment method within 5 to 7 business days of Acme receiving the item.
  • Gift cards cannot be refunded.
  • Damaged items: report within 48 hours of delivery with a photo, and pickup is free. Other return pickups cost Rs 99.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

against the practice bot

How this is checked: When you submit, your tests run against 4 versions of RefundBot: 2 correct and 2 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

How to test chatbot answers that change every time without exact text matching

AI Testing for QA EngineersBeginnerWhy exact match fails on AI

How it works

An assertion that compares a whole reply with one remembered sentence will fail on the next correct reply, because the wording moves. Teams then mark the test flaky and stop trusting it, and a real fault slips through unnoticed.

The robust approach checks what must be true of every correct reply: here, the return window of 30 days must appear. A second kind of check guards what must never appear, such as another customer's email address or phone number. Both are properties, and Python's re module is enough to express them.

Grading uses several builds you cannot see. Correct builds word their answers differently; faulty builds carry a planted defect. A good test passes on all the correct ones and fails on every faulty one, which is the standard every coded test in this course is held to.

About the assignment

Run the colleague's exact-match test to see it fail, then rewrite it to check the fact that matters, and add a test that fails if a reply leaks personal data.

What you will do

  1. Click Run first and read why the starter fails on a correct bot.
  2. Replace the whole-sentence comparison with a check for the fact itself.
  3. Write the leak test with patterns for an email address and for an Indian phone number.

Common mistakes

  • Matching any number

    A pattern that accepts any number of days would pass a bot that gives the wrong window. Check for the right one.

  • Leak patterns that are too narrow

    Phone numbers appear with and without spaces or a country code. Test your pattern on more than one format.

In a real test suite

Property checks are the everyday workhorse of AI testing. They are cheap, they run on every build, and when written well they rarely flake, which leaves expensive tools such as LLM judges for the questions code cannot decide.

Related lessons

Next lessonOne run is not evidenceAll lessons in the courseAI Testing for QA Engineers