Skip to content
CodeKruPro

Same question, different answers

1/99

Module · Why exact match fails on AI

beginner

about 15 min

What this lesson teaches

The idea

A chatbot built on AI writes every reply fresh. Ask it the same question twice and the wording can change, even when the facts in the reply stay the same.

Why a tester cares

Many tests compare a reply with one expected sentence, word for word. On an AI chatbot that test fails on correct replies just because the wording moved. To test AI, you check the facts in a reply, not its exact words.

Words you will meet

Chatbot:
a program you talk to in plain language. RefundBot answers Acme Store customers.
Language model (LLM):
the AI that writes a chatbot's replies. ChatGPT and Claude are language models.
Console:
the chat box in the practice area, where you send messages to the bot. Every message and reply is saved.
Exact-match test:
a test that passes only when the reply equals one expected sentence, word for word.

What you will do

  1. 1Predict
  2. 2Ask five times
  3. 3Count and answer
  4. 4Submit
Jump to practice

Acme Store's support chatbot, RefundBot, answers customer questions about returns. In this lesson you ask it the same question five times and look closely at what comes back.

Your task

  1. Predict how many different replies you will get. Any guess is fine: you will compare it with what really happens.
  2. In the RefundBot console, click the suggested question and press Send. Do this five times, so every message uses exactly the same words.
  3. Count the different wordings. Replies that match word for word count once: five identical replies is 1, five all different is 5.
  4. Answer the two questions: what changed between the replies, and what a test should check.

The refund policy below is what RefundBot is supposed to follow. You will need it in later lessons.

Acme Store refund policy

  • Most items can be returned within 30 days of delivery for a full refund.
  • Refunds reach the original payment method within 5 to 7 business days of Acme receiving the item.
  • Gift cards cannot be refunded.
  • Damaged items: report within 48 hours of delivery with a photo, and pickup is free. Other return pickups cost Rs 99.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Before you start: predict

Prediction: if you send the same question five times, how many different replies will you get?

How this is checked: Not marked right or wrong. You only have to answer it before you start.

2

RefundBot

No messages yet. Start with a suggested question below.

3

Your answer

Replies that match word for word count once. Five identical replies is 1; five all different is 5.

How this is checked: When you submit, the server counts the different wordings in your own RefundBot log and compares that with your number. It ignores capital letters and extra spaces. Your result shows how it counted.

4

Question

Which statement matches what you saw?

How this is checked: One answer is right. Compare the number of days in each of your replies.

5

Question

So what should an automated test of this question check?

How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the others pass.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Why chatbot answers change every time you ask, and what that means for testing

AI Testing for QA EngineersBeginnerWhy exact match fails on AI

How it works

A language model does not look up a stored reply. It writes each answer word by word, choosing from likely next words, and a little randomness in that choice means two requests with identical text can come back worded differently. Products usually keep that randomness on, because fixed wording sounds robotic to customers.

For a tester this changes the unit of evidence. With a normal form or API, one request tells you how the system behaves. With a chatbot, one reply is a sample, and the question becomes what stays the same across many samples. In a refund bot, the window in days should never move even when every sentence around it does.

About the assignment

You predict how much RefundBot will vary, then ask it the same refund-window question several times in the console and compare what came back. The point is to see the difference between wording that changes and facts that must not.

What you will do

  1. Commit to a prediction first. Writing it down before you look is what makes the result stick.
  2. Use the suggestion so every message is identical, and keep the policy panel open while you read.
  3. Count wordings that differ, then judge separately whether the facts in them ever differed.

Common mistakes

  • Counting near-identical replies as one

    A reply that changes only a greeting or a full stop is still a different wording. Compare the text exactly, not the gist.

  • Treating variation as a bug

    Different phrasing is expected behaviour. It only becomes a defect when a fact, a decision or a promise changes.

In a real test suite

Teams that test chatbots stop writing tests that compare whole sentences. They ask the same question many times and check properties of every reply instead: the right number, no forbidden promise, a sensible length. The rest of Module 1 builds those habits one at a time.

Related lessons

Next lessonSame question, real modelAll lessons in the courseAI Testing for QA Engineers