Read a LangSmith export

84/99

Module · Release, production and operations

advanced

about 40 min

What this lesson teaches

The idea

Teams keep traces in tools such as LangSmith and Langfuse. Testers read their exports to find what really failed and to turn real conversations into test cases.

Why a tester cares

An error usually shows on several steps at once, because a step fails when the step inside it fails. Finding the cause means walking down to where it started.

Words you will meet

Span:: one step in a trace, such as retrieval or the model call, with its start, end and result.
Export:: trace data saved as a file.
Root span:: the outermost step: the whole request.

What you will do

1Read both formats
2Follow each error down
3Build the dataset
4Submit

Jump to practice

Teams keep traces in tools such as LangSmith and Langfuse. A tester has to read their exports: find what actually failed, and turn real conversations into test cases. The two formats differ, but both are trees of spans: a root for the request, with children for retrieval, tool calls, the model call and the parser.

An error usually shows on several spans, because a parent fails when a child fails. To find the cause, start at the root and move to the child whose error caused it. Skip a failed child when a later sibling with the same name succeeded: that step was retried and recovered. Stop when no failed child is left.

Reviewers also leave corrections on bad answers. Each is a ready-made test case: the question from the root span, the documents the retrieve span returned, and the corrected answer as the reference.

Your task

Read both exports into one span shape, print the failing span of each failed trace, and build the dataset. The exports here were written for the course in the shape of real ones. On submit it also runs on exports you have not seen.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Your code

1"""Read two trace exports, find the failing span in each failed trace, and build a dataset from corrections."""
2import json
3 
4langsmith = json.load(open("langsmith_export.json"))
5langfuse = json.load(open("langfuse_export.json"))
6 
7 
8def spans_from_langsmith(export):
9"""One dict per span: id, trace, parent, name, start, error (text or None), input, output."""
10return [
11{"id": r["id"], "trace": r["trace_id"], "parent": r["parent_run_id"], "name": r["name"], "start": r["start_time"],
12"error": r["error"], "input": r["inputs"], "output": r["outputs"]}
13for r in export["runs"]
14]
15 
16 
17def spans_from_langfuse(export):
18# Write this: Langfuse marks a failed observation with level "ERROR" and the message in statusMessage.
19return []
20 
21 
22# 1. Put both exports into the same span shape.
23# 2. For each trace whose root span failed, follow the failure down to the span that caused it.
24# 3. One dataset row per trace a reviewer corrected: id, input (the root's question), context (the ids of the
25#    documents the retrieve span returned), reference (the correction).
26print(json.dumps({"failing": {}, "dataset": []}))
27 

on the exports

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Read LangSmith and Langfuse trace exports: find the failing span and build a dataset

AI Testing for QA EngineersAdvancedRelease, production and operations

How it works

Teams keep traces in tools such as LangSmith and Langfuse, and testers need to read their exports: find what actually failed, and turn real conversations into test cases. The two formats differ, but both are trees of spans: a root for the request, with children for retrieval, tool calls, the model call and the parser.

An error usually shows on several spans, because a parent fails when a child fails. Finding the cause means walking down from the root to the child whose error caused it, skipping a failed child when a later sibling with the same name succeeded, since that step was retried and recovered.

Reviewer corrections are ready-made test cases: the question from the root span, the documents the retrieval span returned, and the corrected answer as the reference.

About the assignment

Read both exports into one span shape, print the failing span of each failed trace, and build a dataset from the corrections.

What you will do

Map each format's fields to the same keys, including how each marks an error.
Group spans by parent and sort children by start time.
Walk failures down from the root, then build one dataset row per corrected trace.

Common mistakes

Stopping at the first error
A retried step that later succeeded is not the cause.
Dropping corrected traces that failed
A trace whose retrieval failed still has a question and a correction. Keep it with empty context.

In a real test suite

Exported traces are where most new eval cases come from in mature teams. A small script like this one turns a week of corrections into a dataset update in minutes.

Read a trace like a bug reportReads a single trace by hand.
Failure rate by segment with SQLQueries a trace log with SQL.

Next lessonFailure rate by segment with SQL All lessons in the courseAI Testing for QA Engineers

Read a LangSmith export

What this lesson teaches

The idea

Why a tester cares

Words you will meet

What you will do

Your task

Sign in to practise

Your code

Read LangSmith and Langfuse trace exports: find the failing span and build a dataset

How it works

About the assignment

Common mistakes

Stopping at the first error

Dropping corrected traces that failed

In a real test suite

Related lessons