Every failure mode needs a control, and the cheapest one that does the job is usually right: a code check, an AI judge, a human review, or a permission limit.
Why a tester cares
Some failures are not test problems at all. If a bot can move money past a limit, no pass rate makes that safe; the fix is to remove the ability.
Words you will meet
Code check:
a test a program can decide exactly, such as "contains 30 days".
LLM judge:
an AI model asked to grade a reply, for qualities such as tone.
Permission boundary:
a limit outside the model on what the bot is allowed to do.
Once you know your failure modes, each one needs a control. Pick the cheapest one that can do the job:
Code check: anything a program can decide exactly. JSON shape, a number, a forbidden word, whether the database changed. Fast, free and never moody.
LLM judge: judgments that need reading, such as tone, relevance or whether an answer contradicts its source. Cheaper than a person, but it is a classifier with its own error rate, as Module 10 shows.
Human review: high-stakes meaning that needs expertise, or questions where nobody has written down what "correct" is yet.
Permission boundary, not a test: irreversible actions. If an agent can delete a database or move money past a limit, no pass rate makes that safe. The fix is to remove the ability, or require approval, outside the model.
Your task
Choose the control for each of the 20 failure modes.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
6The reply is longer than the 400-character chat limit.
7The agent can refund more than Rs 10,000 if persuaded.
8The answer contradicts the policy document it cites.
9A cited document ID does not exist.
10Whether a new consumer-law rule changes the refund policy.
11The agent emails every customer on the list at once.
12The reply does not answer the question that was asked.
13The reply is in English when the customer wrote in Hindi.
14A reply mentions a competitor by name.
15The agent can change a customer's bank details.
16A new prompt makes replies sound less like the brand.
17The stated return window is not 30 days.
18The bot claims a refund is done but the database is unchanged.
19An answer gives medical advice it should refer elsewhere.
20The coding agent can push straight to the main branch.
How this is checked: Each row has a right label. At least 16 of 20 must match. For the full score, a harder check: a few key rows must all be right.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
When to use an LLM judge vs code checks, human review or a permission boundary
AI Testing for QA Engineers·Intermediate·Read before you measure
How it works
Each failure mode needs a control, and the cheapest one that can do the job is usually the right one. Code checks decide anything a program can decide exactly. LLM judges handle judgements that need reading, such as tone or contradiction with a source. Human review covers high-stakes meaning that needs expertise.
Some failures are not test problems at all. If an agent can move money past a limit or delete data, no pass rate makes that safe. The fix is a permission boundary outside the model: remove the ability or require approval.
About the assignment
Choose the control for each of twenty failure modes.
What you will do
1Ask first whether a program could decide the failure exactly.
2Keep judges for qualities that need reading but not expertise.
3Send irreversible or high-harm actions to a permission boundary.
Common mistakes
Reaching for a judge first
Judges cost money and make mistakes of their own. Code is cheaper when it can decide.
Testing an action you could forbid
If the action should never happen, the control is to make it impossible.
In a real test suite
A one-page map of failure modes to controls is a common release document. Lesson 23 builds one for a real brief, and Module 13 shows why limits belong in the API, not in the prompt.
Write the choice down per mode, with a one-line reason. A control table that says why each risk got its control survives review by product, security and support teams, and makes later budget cuts easier to argue.
