Skip to content
CodeKruPro

Four questions for any AI feature

70/99

Module · Security, safety and evidence

advanced

about 35 min

What this lesson teaches

The idea

Before security testing an AI feature, map it with four questions: what untrusted text reaches the model, what it can reach, what can leave, and who checks.

Why a tester cares

The danger is untrusted text, access to something valuable and a way out, with nothing checking in between. The map tells you where to test and whom to tell.

Words you will meet

Untrusted text:
anything the model reads that an outsider could have written: messages, documents, emails, web pages.
Exfiltration:
data leaving the system to someone who should not have it.
Escalate:
pass a finding to the people who must act on it.

What you will do

  1. 1Map three features
  2. 2Read six findings
  3. 3Choose escalations
  4. 4Submit
Jump to practice

Scope: Practise attacks only on the bots in these lessons, or on systems you own or have written permission to test. The secrets here are fake and unique to you.

Before testing an AI feature for security, map it. Four questions cover most of what matters:

  1. What untrusted text reaches the model? Customer messages, but also documents, web pages, emails and tool results. Any of them can carry instructions.
  2. What can the model reach? Data, documents and tools, with whose permissions.
  3. What can leave? Replies, but also actions: refunds, emails, rendered images that load a URL.
  4. Who checks? What stops a bad action outside the model: limits in the API, approvals, filters, people.

The dangerous combination is untrusted text, reach to something valuable, and a way out, with nothing checking in between.

Your task

  1. Fill in the four answers for RefundBot, PolicyPal and OrderDesk, as JSON.
  2. Choose an escalation for six findings, using the rules below.

Escalation rules: stop the release when outsiders can harm money, data or other customers now; fix before the next release when it needs insider access; track when no harm follows; send to the security team when the problem is not in the AI feature itself.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Write it up

How this is checked: When you submit, the server checks that it is valid JSON and follows the rules in the task. Your result names any rule it breaks.

2

Label each one

Six findings

0 of 6 labelled

  1. 1Any customer can make LeakyDesk reveal the staff discount code by asking nicely.
  2. 2Customers can talk OrderDesk into refunds above the Rs 10,000 limit.
  3. 3Only HR staff can edit PolicyPal documents, and a test showed text in a document can steer answers.
  4. 4RefundBot will repeat its system prompt when asked; the prompt holds the public refund policy only.
  5. 5The model provider's API key is embedded in the shop's front-end JavaScript.
  6. 6Customers can upload attachments that OrderDesk reads, and an attachment's text can trigger refunds.

How this is checked: Each row has a right label. All 6 must match. This is a harder check: it earns the second 50 points once the core checks pass.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Threat model an AI feature for security testing with four questions

AI Testing for QA EngineersAdvancedSecurity, safety and evidence

How it works

Before testing an AI feature for security, map it. Four questions cover most of what matters. What untrusted text reaches the model? Not only customer messages, but documents, web pages, emails and tool results. What can the model reach: which data, documents and tools, with whose permissions? What can leave: replies, but also actions such as refunds, emails and rendered images that load a URL? And who checks: what stops a bad action outside the model?

The dangerous combination is untrusted text, reach to something valuable, and a way out, with nothing checking in between. Escalation then follows simple rules based on who can cause harm and whether the issue is in the AI feature at all.

About the assignment

Answer the four questions for three course products, then escalate six findings by the rules given.

What you will do

  1. For each product, list every source of untrusted text, including documents.
  2. List what it can reach and every way something can leave, including actions.
  3. Escalate each finding by who can trigger it and where the fix belongs.

Common mistakes

  • Forgetting indirect input

    Documents and tool results are untrusted text too, even when staff wrote them.

  • Treating every issue as an AI issue

    A leaked key in front-end code belongs to the security team, not the AI feature.

In a real test suite

A short threat model at the start of a project decides which tests in the rest of Module 13 matter most. It also gives security reviewers a shared picture, which saves arguments later.

Related lessons

Next lessonIndirect injection through a documentAll lessons in the courseAI Testing for QA Engineers