Bias is tested with pairs: change only one thing that should not matter, such as the customer's name or language, and check the decision stays the same.
Why a tester cares
Results go into an evidence table with the risk, the test, the result and an owner. Testers record evidence; legal judgements belong to others.
Words you will meet
Counterfactual test:
a pair of requests that differ in one detail only.
Bias in an AI feature is tested the same way as everything else: with pairs. Take a request, change only one thing that should not matter, such as the customer's name or the language they write in, and check the decision does not change. These are counterfactual tests, the metamorphic tests of lesson 14 applied to fairness.
Results go into an evidence table: the risk, the test, the result and who owns it. Testers record evidence; they do not give legal verdicts. Whether something breaks a law is for the people whose job that is.
Your task
Write paired tests for names and for Hinglish. On submit they run against a fair build, a build whose decisions depend on the customer's name, and one that answers Hinglish wrongly.
Fill in the evidence table with at least three rows, and no legal conclusions.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
18# Counterfactual pairs: the same request with only a name,or only the language,changed.The decision must
19# not change.
20
against RefundBot
How this is checked: When you submit, your tests run against 3 versions of RefundBot: 1 correct and 2 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.
2
Write it up
How this is checked: When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks. This is a harder check: it earns the second 50 points once the core checks pass.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Counterfactual bias testing for chatbot decisions, with an evidence table and no legal verdicts
AI Testing for QA Engineers·Advanced·Security, safety and evidence
How it works
Bias in an AI feature is tested with pairs. Take a request, change only one thing that should not matter, such as the customer's name or the language they write in, and check that the decision does not change. These are counterfactual tests, the metamorphic relations of lesson 14 applied to fairness.
Results go into an evidence table: the risk, the test, the result and who owns it. Testers record evidence; they do not give legal verdicts. Whether something breaks a law is for the people whose job that is.
About the assignment
Write paired tests for names and for Hinglish across several day counts, and fill an evidence table with at least three complete rows and no legal conclusions.
What you will do
1Keep everything but the name, or the language, identical in each pair.
2Use several name pairs and several day counts on both sides of the window.
3Write what was tested and what happened in each table row.
Common mistakes
Changing more than one thing
If the wording also changes, a different decision proves nothing about names.
Legal words in the table
Terms such as compliant or unlawful are conclusions, not evidence.
In a real test suite
Fairness evidence is often requested by product, legal and compliance teams together. A table that states exactly what was tested makes those conversations faster and more honest.
Choose pairs that a real customer base would produce, including the languages and naming patterns of your users. Generic name lists can miss exactly the groups a product serves most.
