A test plan is a set of bets about where the product will fail. When the model changes and the budget is cut in half, the plan has to change too, and a tester has to explain what it now covers and what it no longer does.
Three questions guide the revision:
What does the new model change? Add cases where the early evidence says it is weaker.
What tells us least for its cost? Run it less often, or on a sample.
What can never be cut? Hard-gate checks on money, safety and personal data.
Your task
Revise the plan for the new model and a Rs 4,000 monthly budget, and cost it.
Answer six questions that come up when the plan meets reality.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
What changed since your plan in lesson 23
1. The product moves from acme-chat-2026-05 to acme-chat-2026-10 next month. Early tests: better English
answers, Hinglish answers 8 points worse, and JSON replies wrapped in code fences 1 time in 40.
2. The eval budget is cut from Rs 8,000 to Rs 4,000 a month.
Current monthly costs (runs x cost per run, in rupees)
smoke on every PR 120 x 2 = 240
golden set on prompt PRs 15 x 40 = 600
safety on prompt PRs 15 x 25 = 375
full judge before releases 4 x 300 = 1,200
weekly full judge 4 x 300 = 1,200
weekly human review 4 x 900 = 3,600 (two reviewers, 3 hours each)
load test before releases 4 x 150 = 600
total 7,815
1
Write it up
How this is checked: When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks. Compare it with the model answer after you submit.
2
Write it up
One row per item: runs a month and cost per run. The total is the sum of runs × cost.
How this is checked: When you submit, the server checks that it is valid JSON and follows the rules in the task. Your result names any rule it breaks.
3
Question
How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the core checks pass.
4
Question
How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the core checks pass.
5
Question
How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the core checks pass.
6
Question
How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the core checks pass.
7
Question
How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the core checks pass.
8
Question
How this is checked: One answer is right. This is a harder check: it earns the second 50 points once the core checks pass.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
Revise an AI test plan after a model upgrade and budget cut, and defend what you keep
AI Testing for QA Engineers·Advanced·Release, production and operations
How it works
A test plan is a set of bets about where the product will fail. When the model changes and the budget halves, the plan has to change too, and a tester has to explain what it now covers and what it no longer does.
Three questions guide the revision. What does the new model change? Add cases where early evidence says it is weaker. What tells us least for its cost? Run it less often or on a sample. What can never be cut? Hard-gate checks on money, safety and personal data.
About the assignment
Revise the plan for a new model and a smaller monthly budget, cost it so the rows add up and fit, and answer six questions that come up when the plan meets reality.
What you will do
1Read what the early tests found about the new model.
2Cut or sample the most expensive items that tell you least.
3Keep the safety suite, cost every row, and check the total against the budget.
Common mistakes
Cutting safety to save money
The hard-gate checks are the line the plan defends.
A total that does not match its rows
Reviewers check the arithmetic first. Make it add up.
In a real test suite
Plans written as risks and controls are easy to revise like this. Plans written as lists of activities usually get cut across the board, which removes protection exactly where it matters.
Write down the risks you now accept, with an owner and a review date, so the next person to read the plan knows what was traded away and why.
