Skip to content
CodeKruPro

Metamorphic tests when no oracle exists

14/99

Module · A deterministic harness

beginner

about 35 min

What this lesson teaches

The idea

Sometimes nobody can say what the exact right answer is. Instead, change the question in a way that should not matter, and check the decision stays the same.

Why a tester cares

This gives you tests where no expected answer exists, and catches a bot that changes its decision because of wording.

Words you will meet

Oracle:
a way of knowing the right answer for a test.
Metamorphic test:
a test that checks how two answers relate, instead of checking one answer.
Paraphrase:
the same question in different words.

What you will do

  1. 1Read the two relations
  2. 2Write the paraphrase test
  3. 3Write the irrelevant-text test
  4. 4Submit
Jump to practice

Often you cannot say what the right answer is for every input. A metamorphic test sidesteps that. Instead of checking one answer, it checks a relation between two answers: if I change the input in a way that should not matter, the decision must not change.

Two relations work for almost any AI feature:

  • Paraphrase: the same request in different words gets the same decision.
  • Irrelevant text: adding a sentence that has nothing to do with the request changes nothing.

RefundBot can answer in JSON mode, where its reply carries a field eligible: true or false. Compare that field, not the wording. Wording changes on every call, even on a correct build.

Your task

Write both relations as tests. Some faulty builds only go wrong for some numbers of days, so test several values on both sides of the 30-day window, such as 29, 31 and 60.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

Write it in
Both are graded the same way.
against the practice bot

How this is checked: When you submit, your tests run against 4 versions of RefundBot: 2 correct and 2 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Metamorphic testing for AI chatbots: when there is no oracle for the right answer

AI Testing for QA EngineersBeginnerA deterministic harness

How it works

Sometimes nobody can say what the exact right answer is for every input. Metamorphic testing checks a relation between answers instead: change the input in a way that should not matter, and the decision must stay the same.

Two relations fit almost every AI feature. Paraphrase: the same request in other words gets the same decision. Irrelevant text: adding an unrelated sentence changes nothing. Comparing a structured decision field, not the wording, keeps the tests stable.

About the assignment

Write a paraphrase relation and an irrelevant-text relation over RefundBot's eligibility decision, across several day counts on both sides of the return window.

What you will do

  1. Use JSON mode so each reply carries the decision field.
  2. Choose day counts just inside, just outside and far from the window.
  3. Assert that each pair of decisions match, and that a decision was actually returned.

Common mistakes

  • Comparing wording

    Wording changes on every call even on correct builds. Compare the decision.

  • Testing one number of days

    Some faults only appear near the boundary. Cover several values.

In a real test suite

Metamorphic relations are a standard way to test search, recommendation and ranking systems, and they extend naturally to fairness: change only a name or a language, and the outcome must not move. Lesson 78 uses exactly that.

Relations can also be checked in reverse. Changing something that should matter, such as moving the purchase date past the window, must change the decision. A bot that ignores its input entirely would pass every same-decision relation, so one changing relation keeps the suite honest.

Related lessons

Next lessonBuild a tiny eval harnessAll lessons in the courseAI Testing for QA Engineers