Skip to content
CodeKruPro

Same question, real model

2/99

Module · Why exact match fails on AI

beginner

about 15 min

What this lesson teaches

The idea

A practice bot follows a script. A real model writes every reply itself: sometimes it repeats one answer word for word, sometimes it rewords it, reorders it or adds details nobody asked for.

Why a tester cares

A test that passes on a practice bot can still fail on the real product. Seeing how a real model actually answers shows what your tests must cope with, and that you only find out by asking it several times.

Words you will meet

Practice bot:
a bot whose replies we scripted, like RefundBot, so every learner gets the same behaviour.
Real model:
an AI model from an AI company, such as Claude, running with Acme's instructions.
System prompt:
the instructions a company gives the model before the customer's message. Here it is the refund policy.
Stand-in:
a practice model that answers when real replies are switched off. The chat says when it is used.

What you will do

  1. 1Predict
  2. 2Ask the real model
  3. 3Ask RefundBot
  4. 4Compare
  5. 5Submit
Jump to practice

RefundBot is a practice bot: we wrote its replies in advance, so the lessons work the same for everyone. Here you ask a real AI model the same question and compare the two. The real model has been given Acme's refund policy as its instructions, the way a company sets up its own assistant.

Your task

  1. Predict which will vary more: the real model or RefundBot.
  2. Send the suggested question five times to the real model, then five times to RefundBot.
  3. For each, count the different wordings. Replies that match word for word count once: five identical replies is 1.
  4. Check whether every real-model reply said 30 days.

Either one may turn out to vary more. That is fine: your answers are checked against your own replies, not against what we expected.

Real models cost money to run, so you have 12 real-model messages for this lesson. Use them on the suggested question. RefundBot has no limit.

The line above the real-model chat says where its replies come from: a live call, replies recorded earlier from a real model, or a stand-in while real replies are switched off.

Acme Store refund policy

  • Most items can be returned within 30 days of delivery for a full refund.
  • Refunds reach the original payment method within 5 to 7 business days of Acme receiving the item.
  • Gift cards cannot be refunded.
  • Damaged items: report within 48 hours of delivery with a photo, and pickup is free. Other return pickups cost Rs 99.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Before you start: predict

Prediction: which will vary more across five identical questions?

How this is checked: Not marked right or wrong. You only have to answer it before you start.

2

Real model

Real model12 of 12 messages left in this lesson

No messages yet. Start with a suggested question below.

3

RefundBot

No messages yet. Start with a suggested question below.

4

Your answer

Replies that match word for word count once. Five identical replies is 1; five all different is 5.

How this is checked: When you submit, the server counts the different wordings in your own Real model log and compares that with your number. Your result shows how it counted.

5

Your answer

Count the same way: replies that match word for word count once.

How this is checked: Counted the same way, from your own RefundBot log.

6

Question

Did every real-model reply state the 30-day window?

How this is checked: When you submit, the server reads each of your real-model replies for "30 days". This is a harder check: it earns the second 50 points.

7

Question

What does this mean for a test suite written against the practice bot?

How this is checked: One answer is right. This is a harder check: it earns the second 50 points.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Real LLM vs scripted bot: how their replies vary, and why it matters

AI Testing for QA EngineersBeginnerWhy exact match fails on AI

How it works

Practice bots in this course are scripted with seeded wording, so the same lesson behaves the same way for every learner and every grading run. That makes them fair to grade, but it hides how messy a real model can be: its length, tone and even its facts can drift between calls. It can mislead the other way too: a current model with a tight prompt often repeats one answer word for word for several calls, then changes its structure on the next.

This lesson connects the two worlds. The real-model console uses a system prompt that carries the refund policy, the way a company configures its own assistant, and it tells you whether a reply was a live call, a recording made earlier from a real model, or a stand-in used until real replies are switched on. Calls to real models cost money, so the lesson gives you a small allowance and asks you to spend it on the suggested question.

About the assignment

Predict which source varies more, send the same question to both consoles, and compare the number of distinct replies and whether the real model kept to the policy every time.

What you will do

  1. Spend the real-model allowance on the suggested message only; experiments belong on RefundBot.
  2. Count distinct wordings for each source separately.
  3. Read every real reply against the policy before answering whether all of them stated it.

Common mistakes

  • Judging from the first reply

    One reply from either source proves nothing about variation. Compare the full set of five.

  • Assuming the real model always follows its instructions

    A system prompt is a strong hint, not a guarantee. Read each reply for the fact itself.

  • Expecting the real model to vary more

    Sometimes it varies less than the practice bot. Report what your log shows; a stable run today promises nothing about the next model version.

In a real test suite

Real teams keep both kinds of target: a deterministic mock or recording for fast, repeatable CI, and scheduled runs against the live model to catch drift the mock cannot show. Knowing which one a result came from is the first thing to check when a number looks surprising.

Related lessons

Next lessonYour first pass or fail ruleAll lessons in the courseAI Testing for QA Engineers