Skip to content
CodeKruPro

A one-page test approach for an AI feature

23/99

Module · Read before you measure

intermediate

about 60 min

What this lesson teaches

The idea

Before launching an AI feature, a team needs one page that says how it will be tested: what can go wrong, which layer catches each risk, and what blocks a release.

Why a tester cares

It is also the question that opens most AI testing interviews. A good answer makes decisions instead of listing every possible test.

Words you will meet

Test approach:
a short plan of how a feature will be tested and why.
Release gate:
a result that must pass before the feature can ship.
Agentic feature:
a bot that can take actions, such as creating a pickup or issuing credit.

What you will do

  1. 1Read the brief and incidents
  2. 2Write the page
  3. 3Choose a control per risk
  4. 4Submit
Jump to practice

"How would you test this AI feature?" is the question most AI testing interviews start with, and the document every team needs before a launch. A good answer fits on one page and makes decisions: what can go wrong, which layer catches each failure, what blocks a release, and what testing cannot do.

Below is the brief for RefundBot v2, which can now act (create pickups and give store credit), and eight short incident cards from real and composite cases. Read both before you write.

Your task

  1. Write the test approach on the template. Keep every heading. Under Layers and controls, give each of the brief's eight risks a layer and a control.
  2. Choose the control for each of the eight risks.

The grader checks the structure and that every risk is covered. The quality of your thinking is yours to judge: compare your page with the model answer after you submit. You will revise this page in lesson 88.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
RefundBot v2 launch briefShow
RefundBot v2 launch brief (Acme Store, summer sale)

What changes
- RefundBot will answer in English, Hindi and Hinglish, and will be able to start a return (create a pickup)
  instead of only explaining the policy.
- It reads the customer's order through lookup_order and can call create_pickup and issue_store_credit.
- Store credit is new: up to Rs 2,000 as goodwill, without a person approving it.

Who uses it
- About 40,000 chats a week in sale weeks, 70% on phones, many in Hinglish.
- Support agents see a summary of each chat when a customer asks for a person.

What the business wants
- Fewer chats handed to people (target: from 35% to 20%).
- No increase in wrong refunds or complaints to the consumer helpline.

Known risks the team listed
1. States a wrong policy fact (window, fee, refund time).
2. Answers in the wrong language.
3. Creates a pickup for an order that is not delivered.
4. Gives store credit above Rs 2,000, or to the wrong order.
5. Does not hand off when a customer asks for a person or threatens legal action.
6. Leaks another customer's details from an order lookup.
7. Summary for support agents leaves out what the customer asked for.
8. Sounds rude or dismissive in a complaint.

Eight incident cards

I1 to I4 are reported real cases; I5 to I8 are composites written for this course.

I1Air Canada chatbot (2024)
  1. 1. What happenedA tribunal in British Columbia held Air Canada responsible for its website chatbot telling a customer he could claim a bereavement fare after travel, which the airline's policy did not allow. The airline had to pay the difference.
I2A model that agreed too much (2025)
  1. 1. What happenedOpenAI rolled back a GPT-4o update in April 2025 after it became overly flattering and agreeable. Its own account says offline evaluations and A/B tests looked positive, while some expert testers had said the model felt "off".
I3Replit agent and a production database (2025)
  1. 1. What happenedA Replit coding agent deleted a company's production database during a declared code freeze, then reported misleadingly on what it had done. Replit added separation between development and production afterwards.
I4Kiro and an AWS outage (reported 2026)
  1. 1. What happenedA press report said an AWS coding agent, Kiro, chose to delete and recreate a customer-facing system, causing a long outage. Amazon disputed the report's account. Treat the details as contested.
I5Composite: the stale document
  1. 1. What happenedA composite example, not one company. After a policy change, the old document still ranked first in search, and the bot quoted the old fee for three weeks.
I6Composite: the language gap
  1. 1. What happenedA composite example. A bot's accuracy on English questions was 94%, but on Hinglish questions it was 71%. The overall number hid it because Hinglish was 15% of traffic.
I7Composite: the summary that dropped the ask
  1. 1. What happenedA composite example. Agent hand-off summaries read well, but one in eight left out what the customer actually wanted, so agents asked again.
I8Composite: the healed test
  1. 1. What happenedA composite example. A test-healing tool "fixed" a failing checkout test by asserting the wrong total the app now showed, hiding a real pricing bug for two releases.
1

Write it up

How this is checked: When you submit, the server checks that it is valid Markdown and follows the rules in the task. Your result names any rule it breaks. Compare it with the model answer after you submit.

2

Label each one

Control for each risk in the brief

0 of 8 labelled

  1. 1States a wrong policy fact (window, fee, refund time)
  2. 2Answers in the wrong language
  3. 3Creates a pickup for an order that is not delivered
  4. 4Gives store credit above Rs 2,000 or to the wrong order
  5. 5Does not hand off when a customer asks for a person
  6. 6Leaks another customer's details from an order lookup
  7. 7Agent summary leaves out what the customer asked for
  8. 8Sounds rude or dismissive in a complaint

How this is checked: Each row has a right label. At least 7 of 8 must match. A few key rows must all be right. This is a harder check: it earns the second 50 points once the core checks pass.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Test strategy for an AI feature on one page: risks, layers, release blockers and limits

AI Testing for QA EngineersIntermediateRead before you measure

How it works

The question that opens most AI testing interviews is also the document every team needs before a launch: how will this feature be tested? A good answer fits on one page and makes decisions. It names what can go wrong, which layer catches each risk, what blocks a release, and what testing cannot promise.

Agentic features raise the stakes. When a bot can create pickups or give store credit, some risks need permission boundaries, not just tests, and the page should say so.

About the assignment

Write a one-page test approach for RefundBot v2 on the template, giving each risk in the brief a layer and a control, informed by eight short incident cards.

What you will do

  1. Read the brief and every incident card before writing.
  2. Keep each heading of the template, and fill every section.
  3. Give each risk one layer and one control, and say what blocks release.

Common mistakes

  • A list of test types

    A strategy decides. Say what you will and will not test, and why.

  • No limits section

    Saying what testing cannot catch is what makes the rest credible.

In a real test suite

Lesson 88 revisits this page after a model upgrade and a budget cut. Plans that name risks and controls are easy to revise; plans that list activities are not.

Read the incident cards as evidence of what goes wrong in practice. Each one should change something on your page: a risk you had not listed, a stronger control, or a clearer release blocker.

Related lessons

Next lessonGive the assistant what a tester knowsAll lessons in the courseAI Testing for QA Engineers