Skip to content
CodeKruPro

Mutation testing as review

31/99

Module · Reviewing AI-written tests

intermediate

about 45 min

What this lesson teaches

The idea

Mutation testing makes small deliberate bugs in the code, one at a time, and checks whether any test fails.

Why a tester cares

Each surviving mutant is a real bug your suite would miss. It is the most honest review of a generated suite.

Words you will meet

Mutant:
a copy of the code with one small change, such as > turned into >=.
Killed:
a mutant some test caught.
Survived:
a mutant every test passed, so the suite missed it.

What you will do

  1. 1Run the mutation tester
  2. 2Read the survivors
  3. 3Add tests
  4. 4Submit
Jump to practice

Mutation testing asks the question coverage cannot: if this code were wrong, would a test fail? A tool makes small changes to the code, one at a time, such as < to <= or - to +. Each changed version is a mutant. If your tests fail, the mutant is killed. If they pass, it survived, and you have found a bug your suite would miss.

It is the most honest review of a generated suite, because a suite that asserts nothing kills nothing. In Python the usual tool is mutmut (mutmut run, then mutmut results); Java teams use PIT, and JavaScript teams use Stryker. This lesson uses a small mutation tester with the same idea and a fixed list of mutants, so runs are fast and repeatable.

Your task

Add tests to test_pricing.py until the mutation score is at least 0.8. Run shows which mutants survived. On submit, your tests also face mutants you have not seen, and a completely broken version of the module that a real test must fail on.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

the mutation tester

How this is checked: When you submit, your script runs on 3 sets of data, 2 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Mutation testing to review a test suite: killed and surviving mutants explained

AI Testing for QA EngineersIntermediateReviewing AI-written tests

How it works

Mutation testing asks what coverage cannot: if this code were wrong, would any test fail? A tool makes one small change at a time, such as flipping a comparison or an operator, and runs the suite on each changed version. A failing suite kills the mutant; a passing one lets it survive.

Each survivor is a concrete bug your suite would miss, which makes it the most honest review of a generated suite. Python teams use mutmut, Java teams PIT, and JavaScript teams Stryker; this lesson uses a small mutation tester with a fixed list of mutants so results are repeatable.

About the assignment

Run the mutation tester on the pricing code, read which mutants survived, and add tests until the survivors are killed.

What you will do

  1. Run the tester and read each surviving mutant's change.
  2. For each survivor, write a test with an input where the change would give a different answer.
  3. Re-run until the suite kills the mutants the lesson asks for.

Common mistakes

  • Testing the mutant instead of the rule

    A good new test checks the spec's behaviour, and kills the mutant as a side effect.

  • Ignoring boundaries

    Most survivors live at boundaries, such as exactly the minimum subtotal. Test the edge values.

In a real test suite

Running mutation tests on every commit is slow, so teams run them nightly or on the code a pull request touches, and track the survival rate as a suite quality metric.

Related lessons

Next lessonHeal or rejectAll lessons in the courseAI Testing for QA Engineers