Skip to content
CodeKruPro

Read a trace like a bug report

18/99

Module · Read before you measure

beginner

about 25 min

What this lesson teaches

The idea

A trace records every step of one request: the instructions, the customer's message, any documents fetched, tool results and the final reply.

Why a tester cares

Reading a trace is like reading a bug report with the logs attached. For each claim in a reply you can find where it came from, or see that it came from nowhere.

Words you will meet

Trace:
the full record of one request, step by step.
Retrieval:
fetching documents for the model to read before it answers.
Tool call:
the model asking the app to do something, such as look up an order.

What you will do

  1. 1Open each trace
  2. 2Find each item's source
  3. 3Submit
Jump to practice

When an AI feature gives a wrong answer, the first question is not "is the model bad?" but "where did this come from?". A trace records each step of one request: the system prompt, what the user typed, what was retrieved or what tools returned, and the reply. Reading one is like reading a bug report with the logs attached.

For every claim in a reply there are five possible sources: the customer's input, the system prompt, retrieved text, a tool result, or nothing at all. The last one matters most. A claim with no source is the model filling a gap, and it is where many wrong answers start.

Your task

Read the six traces. For each marked part of a reply, choose where it came from: the customer, the system prompt, retrieved text, a tool result, or nothing. Trace D is worth a second look: its reply has a source, and is still wrong.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account

Six traces

Open a trace to see its steps in order.

ARefundBot: can I still return the kettle?RefundBot
  1. 1. System promptYou are Acme Store's refunds assistant. Answer in at most two sentences, using only the policy you are given.
  2. 2. CustomerOrder AC-48213: can I still return the kettle? It came 12 days ago.
  3. 3. Retrieved policyMost items can be returned within 30 days of delivery for a full refund.
  4. 4. ReplyYes. At 12 days you are inside our 30-day window, so start the return for AC-48213 from My Orders.
BOrderDesk: refund AC-10001OrderDesk
  1. 1. CustomerPlease refund AC-10001.
  2. 2. lookup_order(AC-10001){"status": "delivered", "amount": 1200, "refunded": 0}
  3. 3. issue_refund(AC-10001, 1200){"refund_id": "RF-0001"}
  4. 4. ReplyDone: Rs 1,200 is refunded for AC-10001. You'll see it in your account in 2 days.
CRefundBot: damaged parcel, in HinglishRefundBot
  1. 1. System promptReply in the customer's language. Keep replies to two sentences.
  2. 2. Customermera parcel toota hua aaya, kya karu?
  3. 3. Retrieved policyDamaged items: report within 48 hours of delivery with a photo, and pickup is free.
  4. 4. ReplyMaaf kijiye! 48 ghante ke andar photo ke saath report kijiye. Pickup free hai.
DPolicyPal: annual leavePolicyPal
  1. 1. EmployeeHow many days of annual leave do I get?
  2. 2. Retrieved documents1. leave-2023 (replaced in 2025): "...18 days of paid annual leave a year." 2. leave-2025: "...24 days of paid annual leave a year."
  3. 3. ReplyYou get 18 days of paid annual leave a year. [leave-2023]
EOrderDesk: where is my order?OrderDesk
  1. 1. CustomerWhere is AC-10003?
  2. 2. lookup_order(AC-10003){"status": "shipped", "expected_delivery": "Friday"}
  3. 3. ReplyYour order AC-10003 was delivered yesterday.
FRefundBot: paying for pickupRefundBot (hidden check)
  1. 1. CustomerHow much is pickup and can I pay by UPI?
  2. 2. Retrieved policyOther return pickups cost Rs 99. Damaged items are picked up free.
  3. 3. ReplyPickup costs Rs 99, and you can pay by UPI or card at the door.
1

Label each one

Where did each part of the reply come from?

0 of 12 labelled

  1. 1Trace AThe "30-day window" in the reply
  2. 2Trace AThe order number AC-48213
  3. 3Trace BThe amount Rs 1,200
  4. 4Trace B"You'll see it in your account in 2 days"
  5. 5Trace CThe reply being two sentences long
  6. 6Trace C"48 ghante" (48 hours)
  7. 7Trace DThe number 18
  8. 8Trace DThe citation [leave-2023]
  9. 9Trace E"was delivered yesterday"
  10. 10Trace EThe order number AC-10003
  11. 11Trace FThe price Rs 99
  12. 12Trace F"you can pay by UPI or card at the door"

How this is checked: Each row has a right label. At least 8 of 10 key rows must be right. A few key rows must all be right. For the full score, a harder check: all 12 must match.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

How to read an LLM trace: tracing each claim in a reply back to its source

AI Testing for QA EngineersBeginnerRead before you measure

How it works

A trace records every step of one request: the system prompt, the user's message, retrieved text, tool results and the final reply. Reading it is like reading a bug report with the logs attached.

For each claim in a reply there are only a few possible sources: the input, the system prompt, retrieved text, a tool result, or nothing. A claim with no source is the model filling a gap, and many wrong answers start there. A claim can also have a source and still be wrong, when the source itself is out of date.

About the assignment

Open six traces and, for each marked part of a reply, choose where it came from.

What you will do

  1. Open the trace and read every step before the reply.
  2. For each item, find the earliest step that contains the claim.
  3. Mark a claim as having no source only when no step contains it.

Common mistakes

  • Assuming the reply invented it

    Check the retrieved text and tool results first. Many odd claims are copied from a step.

  • Stopping at the first match

    A claim may appear in a later step only because an earlier step supplied it. Look for the first.

In a real test suite

Trace reading is the core skill of error analysis. Observability tools such as LangSmith and Langfuse display the same steps; lesson 84 reads their exports directly.

Keep notes while you read. A short line per trace, naming the claim and its source, is the raw material that error analysis turns into a taxonomy in the next two lessons.

Related lessons

Next lessonNote what went wrong, and where it first went wrongAll lessons in the courseAI Testing for QA Engineers