Skip to content
CodeKruPro

What the agent sees

33/99

Module · Agents in the browser: Playwright MCP and test agents

intermediate

about 25 min

What this lesson teaches

The idea

Browser agents read a web page through its accessibility tree, the same information a screen reader uses, not through the pixels you see.

Why a tester cares

A button with no label or role is nearly invisible to an agent. Testing for agents and testing for accessibility are the same job.

Words you will meet

Agent:
an AI that takes actions, here clicking and typing in a browser.
Accessibility snapshot:
a list of the page's elements with their roles and names.
Role:
what an element is, such as button, link or textbox.

What you will do

  1. 1Look at the screenshot
  2. 2Read the snapshot
  3. 3Label each element
  4. 4Submit
Jump to practice

Browser agents, such as an AI assistant driving Playwright through MCP, mostly do not look at screenshots. They read the page's accessibility snapshot: a tree of roles and names, the same information a screen reader uses. If an element has no role or no name there, the agent has to guess, or cannot find it at all.

That makes accessibility testing and agent testing the same job. A control drawn on a canvas, an icon button with no label, or a "button" that is really a styled div all look fine in a screenshot.

Your task

Below is ShopLab's cart in a build with accessibility defects: a screenshot, part of its HTML, and the accessibility snapshot Playwright produced for it. For each element, decide what an agent gets from the snapshot.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
What a person seesShowShopLab cart with two items, a coupon box with an Apply button, a summary showing Rs 747, and an orange Place order bar.
Part of the HTMLShow
<h1>Your cart</h1>
<table>
<caption class="sr">Items in your cart</caption>
<thead>
<tr>
<th>Item</th>
<th>Quantity</th>
<th>Price</th>
<th>
<span class="sr">Actions</span>
</th>
</tr>
</thead>
<tbody>
<tr data-testid="cart-line" data-sku="MUG">
<td>Coffee mug</td>
<td>
<div class="qty" tabindex="0" onclick="this.nextElementSibling.hidden=!this.nextElementSibling.hidden">2 ▾</div>
<div hidden="">
<div class="opt" onclick="document.getElementById('q-MUG').value=1;document.getElementById('f-MUG').submit()">1</div>
<div class="opt" onclick="document.getElementById('q-MUG').value=2;document.getElementById('f-MUG').submit()">2</div>
<div class="opt" onclick="document.getElementById('q-MUG').value=3;document.getElementById('f-MUG').submit()">3</div>
<div class="opt" onclick="document.getElementById('q-MUG').value=4;document.getElementById('f-MUG').submit()">4</div>
<div class="opt" onclick="document.getElementById('q-MUG').value=5;document.getElementById('f-MUG').submit()">5</div>
<div class="opt" onclick="document.getElementById('q-MUG').value=6;document.getElementById('f-MUG').submit()">6</div>
<div class="opt" onclick="document.getElementById('q-MUG').value=7;document.getElementById('f-MUG').submit()">7</div>
<div class="opt" onclick="document.getElementById('q-MUG').value=8;document.getElementById('f-MUG').submit()">8</div>
<div class="opt" onclick="document.getElementById('q-MUG').value=9;document.getElementById('f-MUG').submit()">9</div>
<div class="opt" onclick="document.getElementById('q-MUG').value=10;document.getElementById('f-MUG').submit()">10</div>
</div>
<form id="f-MUG" method="post" action="/r/9bc6d118395e65268f34b2b722614a47/shop/cart/qty">
<input type="hidden" name="sku" value="MUG" style="">
<input type="hidden" id="q-MUG" name="qty" value="2" style="">
</form>
</td>
<td data-testid="line-total">Rs 598</td>
<td>
<form method="post" action="/r/9bc6d118395e65268f34b2b722614a47/shop/cart/remove">
<input type="hidden" name="sku" value="MUG" style="">
<button type="submit" class="icon">
<svg width="14" height="14" viewBox="0 0 14 14" aria-hidden="true">
<path d="M2 4h10M5 4V2h4v2M3 4l1 8h6l1-8" stroke="#fff" fill="none">
</path>
</svg>
</button>
</form>
</td>
</tr>
<tr data-testid="cart-line" data-sku="NOTE">
<td>Notebook</td>
<td>
<div class="qty" tabindex="0" onclick="this.nextElementSibling.hidden=!this.nextElementSibling.hidden">1 ▾</div>
<div hidden="">
<div class="opt" onclick="document.getElementById('q-NOTE').value=1;document.getElementById('f-NOTE').submit()">1</div>
<div class="opt" onclick="document.getElementById('q-NOTE').value=2;document.getElementById('f-NOTE').submit()">2</div>
<div class="opt" onclick="document.getElementById('q-NOTE').value=3;document.getElementById('f-NOTE').submit()">3</div>
<div class="opt" onclick="document.getElementById('q-NOTE').value=4;document.getElementById('f-NOTE').submit()">4</div>
<div class="opt" onclick="document.getElementById('q-NOTE').value=5;document.getElementById('f-NOTE').submit()">5</div>
<div class="opt" onclick="document.getElementById('q-NOTE').value=6;document.getElementById('f-NOTE').submit()">6</div>
<div class="opt" onclick="document.getElementById('q-NOTE').value=7;document.getElementById('f-NOTE').submit()">7</div>
<div class="opt" onclick="document.getElementById('q-NOTE').value=8;document.getElementById('f-NOTE').submit()">8</div>
<div class="opt" onclick="document.getElementById('q-NOTE').value=9;document.getElementById('f-NOTE').submit()">9</div>
<div class="opt" onclick="document.getElementById('q-NOTE').value=10;document.getElementById('f-NOTE').submit()">10</div>
</div>
<form id="f-NOTE" method="post" action="/r/9bc6d118395e65268f34b2b722614a47/shop/cart/qty">
<input type="hidden" name="sku" value="NOTE" style="">
<input type="hidden" id="q-NOTE" name="qty" value="1" style="">
</form>
</td>
<td data-testid="line-total">Rs 149</td>
<td>
<form method="post" action="/r/9bc6d118395e65268f34b2b722614a47/shop/cart/remove">
<input type="hidden" name="sku" value="NOTE" style="">
<button type="submit" class="icon">
<svg width="14" height="14" viewBox="0 0 14 14" aria-hidden="true">
<path d="M2 4h10M5 4V2h4v2M3 4l1 8h6l1-8" stroke="#fff" fill="none">
</path>
</svg>
</button>
</form>
</td>
</tr>
</tbody>
</table>
<form id="cf" method="post" action="/r/9bc6d118395e65268f34b2b722614a47/shop/cart/coupon" style="margin:16px 0">
<input name="code" placeholder="Coupon code" autocomplete="off" style=""> <canvas id="apply" width="64" height="34" style="vertical-align:middle;cursor:pointer">
</canvas>
<script>/* draws "Apply" on the canvas and submits the form on click */</script>
</form>
<p role="status" data-testid="message">
</p>
<dl aria-label="Order summary">
<dt>Subtotal</dt>
<dd data-testid="subtotal">Rs 747</dd>
<dt>Discount</dt>
<dd data-testid="discount">Rs 0</dd>
<dt>Shipping</dt>
<dd data-testid="shipping">Free</dd>
<dt>
<strong>Total</strong>
</dt>
<dd data-testid="total">
<strong>Rs 747</strong>
</dd>
</dl>
<div style="margin-top:16px">
<form id="po" method="post" action="/r/9bc6d118395e65268f34b2b722614a47/shop/order">
</form>
<div class="btn" onclick="document.getElementById('po').submit()">Place order</div>
</div>
What an agent sees: the accessibility snapshotShow
- main:
  - heading "Your cart" [level=1]
  - table "Items in your cart":
    - caption: Items in your cart
    - rowgroup:
      - row "Item Quantity Price Actions":
        - columnheader "Item"
        - columnheader "Quantity"
        - columnheader "Price"
        - columnheader "Actions"
    - rowgroup:
      - row "Coffee mug 2 ▾ Rs 598":
        - cell "Coffee mug"
        - cell "2 ▾"
        - cell "Rs 598"
        - cell:
          - button
      - row "Notebook 1 ▾ Rs 149":
        - cell "Notebook"
        - cell "1 ▾"
        - cell "Rs 149"
        - cell:
          - button
  - textbox "Coupon code"
  - status
  - term: Subtotal
  - definition: Rs 747
  - term: Discount
  - definition: Rs 0
  - term: Shipping
  - definition: Free
  - term:
    - strong: Total
  - definition:
    - strong: Rs 747
  - text: Place order
1

Label each one

What does the snapshot give an agent?

0 of 8 labelled

  1. 1The "Your cart" heading
  2. 2The coupon code box
  3. 3The Apply control
  4. 4The bin icons that remove an item
  5. 5The quantity selectors ("2 ▾")
  6. 6Place order
  7. 7The total amount
  8. 8The cart's item rows

How this is checked: Each row has a right label. At least 6 of 8 must match. For the full score, a harder check: all 8 must match.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Accessibility snapshot for AI browser agents: why agents miss unlabelled controls

AI Testing for QA EngineersIntermediateAgents in the browser: Playwright MCP and test agents

How it works

Browser agents, such as an assistant driving Playwright through MCP, mostly read the page's accessibility snapshot rather than its pixels. The snapshot is a tree of roles and names, the same information a screen reader uses.

An element without a role or a name is nearly invisible there. A control drawn on a canvas, an icon button with no label, or a styled div acting as a button all look fine in a screenshot, and an agent has to guess at them or cannot find them at all. That makes accessibility testing and agent testing the same job.

About the assignment

Compare a screenshot of ShopLab's cart with its HTML and accessibility snapshot, and label each element by whether an agent could find and use it.

What you will do

  1. Look at the screenshot and list the controls a person sees.
  2. Find each one in the snapshot and note its role and name.
  3. Label the elements that are missing a role, a name, or both.

Common mistakes

  • Judging from the screenshot

    An element that looks like a button may not be one in the snapshot.

  • Treating any text as a name

    Nearby text is not the element's accessible name unless it is associated with it.

In a real test suite

Fixing these defects helps disabled users and agents at once. Teams increasingly add accessibility snapshot checks to CI for exactly that reason.

You can inspect the same tree yourself: Playwright can print an element's accessibility snapshot, and browser developer tools show each element's computed role and name. Checking a new page this way takes minutes and catches problems before an agent or a screen reader user does.

Related lessons

Next lessonBe the agentAll lessons in the courseAI Testing for QA Engineers