After a bad release, the first question is which part got worse. One SQL query over the stored traces answers it, grouped by intent, model version and day.
Why a tester cares
Two details decide whether the numbers are right: leave out rows nobody judged, and avoid whole-number division, which turns every rate into zero.
Words you will meet
SQL:
the language for querying databases.
Group by:
compute a result separately for each group of rows.
Integer division:
dividing whole numbers and dropping the fraction, so 3 / 4 gives 0.
Production traces usually end up in a database, and the first question after a bad release is "which part got worse?". One SQL query answers it: group the pass rate by the things that could matter.
The calls table has one row per judged conversation: day, intent, model_version and passed (1 or 0). Some rows have passed NULL: they were never judged and must not count either way.
Your task
Write query.sql: select intent, model_version, day and pass_rate, one row per combination. Run loads the dump into SQLite and shows your results.
Name the segment that regressed. On submit your query also runs on a database you have not seen.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.
2
Question
How this is checked: One answer is right.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
SQL query for LLM pass rate by segment: intent, model version and day
AI Testing for QA Engineers·Advanced·Release, production and operations
How it works
Production traces usually end up in a database, and the first question after a bad release is which part got worse. One SQL query answers it: group the pass rate by the things that could matter, such as intent, model version and day.
Two details decide whether the numbers are right. Unjudged rows, where the pass column is empty, must not count either way. And integer division in SQLite turns a ratio of whole numbers into zero, which looks like a catastrophic regression that never happened.
About the assignment
Write a query that returns the pass rate for every intent, model and day, run it on the dump, and name the segment that regressed.
What you will do
1Select the four columns the runner expects, in order.
2Group by all three dimensions.
3Leave unjudged rows out of the rate, and read across versions for the same intent and days.
Common mistakes
Grouping by one column
A rate per intent hides which model version and day caused it.
Counting unjudged rows
Rows that were never judged are not failures. Leave them out.
In a real test suite
A saved query like this becomes the first panel on a release dashboard. When a gate blocks, the segment table shows where to look before anyone opens a single trace.
Keep queries in version control next to the eval code, so a schema change breaks a test instead of a dashboard someone trusts.
