Skip to content
CodeKruPro

Test the chat window

68/99

Module · Testing agents, chat UIs and extractors

advanced

about 50 min

What this lesson teaches

The idea

Most AI features reach users through a chat window, and many of its bugs have nothing to do with the model: a cut-off reply shown as complete, Regenerate showing the old answer, history lost on reload.

Why a tester cares

These are ordinary browser tests. Waiting on each message's state, not on a fixed time, keeps them stable.

Words you will meet

Playwright:
a tool for writing tests that drive a real browser.
Message state:
whether a reply is still streaming, done, stopped or errored.
Flaky test:
a test that sometimes passes and sometimes fails with nothing changed.

What you will do

  1. 1Learn the markup
  2. 2Test the stream
  3. 3Test Stop, Regenerate and reload
  4. 4Test bad input
Jump to practice

Most AI features reach users through a chat window, and many of its bugs have nothing to do with the model: a stream that stops halfway and is shown as complete, a Regenerate button that shows the old answer, history that vanishes on reload, a double Enter that sends the question twice. These are browser tests.

ShopLab's help page has a chat widget. Its markup, for your locators:

  • The conversation is a list with role log. Each message is an li with data-role="user" or data-role="assistant".
  • An assistant message has data-state: streaming, then done, stopped or error.
  • Buttons: Send, Stop (while streaming) and Regenerate (after a reply). Errors appear in an element with role alert.
  • Rules: replies about shipping state the Rs 500 threshold and end with a full stop; empty input shows "Type a message first."; input over 2,000 characters is refused as too long.

Your task

Test the six states users hit. On submit your tests run against working widgets and widgets with one bug each.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

Write it in
Both are graded the same way.
against ShopLab's help page

How this is checked: When you submit, your tests run against 5 versions of the ShopLab store: 1 correct and 4 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Test a chatbot chat window UI with Playwright: streams, Stop, Regenerate and history

AI Testing for QA EngineersAdvancedTesting agents, chat UIs and extractors

How it works

Most AI features reach users through a chat window, and many of its bugs have nothing to do with the model. A stream that stops halfway is shown as complete, Regenerate shows the old answer, history vanishes on reload, and a double Enter sends the question twice. These are browser tests.

ShopLab's help widget exposes the state of each message through attributes, such as a role for who wrote it and a state that moves from streaming to done, stopped or error. Tests that wait on those states are stable, where tests that wait a fixed time are flaky.

About the assignment

Write Playwright tests for the chat widget's stream, Stop, Regenerate, reload and bad-input behaviour, so they pass on the correct widget and fail on each broken one.

What you will do

  1. Locate messages by role and state from the markup on the page.
  2. Wait for a reply to reach its final state before asserting its text.
  3. Test Stop, Regenerate, reload and a double send, each in its own test.

Common mistakes

  • Fixed waits

    A sleep is either too short or too slow. Wait for the state attribute instead.

  • Checking only the text

    A cut-off stream can look like a short answer. Check the final state.

In a real test suite

Chat widget bugs are some of the most visible to customers, and they are cheap to test once the widget exposes its states. Teams often ask developers to add such attributes for exactly this reason.

Related lessons

Next lessonScore an extractorAll lessons in the courseAI Testing for QA Engineers