An AI feature is still software. Plenty of its bugs have nothing to do with the model: authentication, input limits, timeouts and broken connections fail the same way they do anywhere else. Knowing which kind of fault you are looking at tells you who should fix it and how to test it.
Your task
In the console, make three ordinary software faults happen and keep them in your log:
send a message without the API key (switch it off in the console options);
send a message that is far too long (the filler button adds 25,000 characters);
turn on streaming and send a normal question. This build's stream drops halfway, so watch how the reply ends.
Sort the eight symptoms below into software faults and model behaviour.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
No messages yet. Start with a suggested question below.
2
Label each one
Where does each symptom come from?
0 of 8 labelled
1The reply stops mid-sentence because the streaming connection dropped.
2The bot says returns are allowed for 14 days when the policy says 30.
3Every request fails with 401 after the API key was rotated.
4The bot invents a "premium returns" programme that Acme does not have.
5Long pasted messages are rejected with a 400 error.
6The chat history disappears when the customer reloads the page.
7Asked in English, the bot sometimes answers in Hindi.
8Two customers who send at the same moment get each other's replies.
How this is checked: Each row has a right label. All 8 must match. This is a harder check: it earns the second 50 points once the core checks pass.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
AI feature bugs that are not AI bugs: keys, input limits, timeouts and broken streams
AI Testing for QA Engineers·Beginner·Why exact match fails on AI
How it works
Much of what breaks in an AI feature is ordinary software. A missing API key, an oversized request, a timeout and a stream that drops halfway fail the same way they do in any web service, and they are fixed by the same people with the same tools.
Sorting faults correctly saves time. A model behaviour, such as a hedged or wrong answer, needs prompt, data or evaluation work. A software fault needs a status code, an error handler or a retry. Mislabelling one as the other sends the bug to the wrong team.
Streaming deserves special attention. A reply that stops early can look complete on screen, so the reliable signal is how the response finished, not how the text reads.
About the assignment
Trigger three ordinary faults from the console and keep them in your log, then sort eight symptoms into software faults and model behaviour.
What you will do
1Switch the API key off in the console options and send a message.
2Use the filler button to send an oversized message and read the status.
3Turn streaming on, send a normal question, and watch how the reply ends.
Common mistakes
Reading only the text
Status codes and finish reasons are the evidence for software faults. Look at them, not just the words.
Calling every odd reply a model problem
An empty reply with a success status is a contract bug, not a personality quirk.
In a real test suite
Incident reviews of AI features often find that the expensive outage was a plain API fault the team never tested, because everyone was focused on answer quality. Module 3 turns these clauses into a contract test suite.
