Skip to content
CodeKruPro

Unseen attacks, then a regression test

77/99

Module · Security, safety and evidence

advanced

about 45 min

What this lesson teaches

The idea

Teams often answer an injection finding with a list of blocked phrases. The next attacker rewords.

Why a tester cares

A sturdier control checks what leaves: the system knows its own secret, so it can stop it going out in any disguise, plain, spaced out, reversed or encoded.

Words you will meet

Blocklist:
a list of phrases to refuse.
Output control:
a check on what the bot is about to send, not on what it received.
Encoding:
rewriting text in another form, such as base64, that hides it from simple filters.

What you will do

  1. 1Run the keyword filter
  2. 2Move the control to the output
  3. 3Handle encodings
  4. 4Submit
Jump to practice

Scope: Practise attacks only on the bots in these lessons, or on systems you own or have written permission to test. The secrets here are fake and unique to you.

Teams often answer a prompt-injection finding with a blocklist: "ignore previous instructions", "secret code". The next attacker rewords. A static list of attacks is not a defence, and neither is a test suite made only of the attacks you already know.

A sturdier control looks at what leaves. The system knows its own secret, so an output check can stop the secret leaving, whether it comes out plain, spaced out, reversed, in base64 or in hex. Input checks are still useful for logging, but they should not block ordinary questions.

Your task

Replace the keyword filter in guard.py. Run shows which recorded attacks it missed and which normal questions it blocked. On submit it also faces attacks you have not seen, including encoded leaks and an injection through a document.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

on recorded LeakyDesk exchanges

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

2

Write it up

How this is checked: Not marked for content: it needs at least 30 words.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Output filter against encoded prompt injection leaks, with a regression test for unseen attacks

AI Testing for QA EngineersAdvancedSecurity, safety and evidence

How it works

Teams often answer an injection finding with a blocklist of phrases. The next attacker rewords. A static list of attacks is not a defence, and neither is a test suite made only of attacks already seen.

A sturdier control looks at what leaves. The system knows its own secret, so an output check can stop it leaving, whether it comes out plain, spaced out, reversed, in base64 or in hex. Input checks remain useful for logging, but should not block ordinary questions.

About the assignment

Replace a keyword filter with an output control that blocks every visible attack and allows every normal question, then holds on unseen and encoded attacks on submit, and write a short bug report.

What you will do

  1. Run the keyword filter and read which attacks it misses.
  2. Block replies that contain the secret, then the secret in common transformations.
  3. Keep the input check permissive, so normal questions are never blocked.

Common mistakes

  • Blocking everything

    A filter that refuses all replies passes the attack count and fails users.

  • Decoding only one format

    Attackers use several encodings. Normalise the reply in several ways before checking.

In a real test suite

Output checks for secrets and personal data are one of the few injection controls that do not depend on predicting the attack. Teams pair them with keeping secrets out of prompts altogether.

Keep the recorded attacks in a file, as this lesson does, and add every new bypass to it. The control is what protects users; the file is what proves, on every change, that the control still holds against everything seen so far.

Related lessons

Next lessonBias checks and an evidence tableAll lessons in the courseAI Testing for QA Engineers