Skip to content
CodeKruPro

Recall@k, Precision@k and MRR by hand

55/99

Module · Testing RAG and hallucination

intermediate

about 30 min

What this lesson teaches

The idea

Retrieval is tested on its own, using document IDs: for each question you know which documents are relevant and which the system returned.

Why a tester cares

Three standard scores measure it, and the edge case that breaks most code is a question that returns nothing.

Words you will meet

Recall@k:
the share of relevant documents found in the top k results.
Precision@k:
the share of the top k results that are relevant.
MRR:
mean reciprocal rank: how near the top the first relevant document appears.

What you will do

  1. 1Implement the metrics
  2. 2Run on the queries
  3. 3Handle empty results
  4. 4Submit
Jump to practice

Retrieval is tested separately from answers, with metrics computed from document IDs. For each query you know which documents are relevant, and you have the ranked list the retriever returned:

  • Recall@k: the share of the relevant documents that appear in the top k.
  • Precision@k: the share of the top k slots that hold a relevant document. Divide by k, even when fewer than k came back.
  • MRR (mean reciprocal rank): 1 divided by the position of the first relevant document, averaged over queries; 0 when none was returned.

Your task

Implement the three functions. The script averages them over the queries at k = 3. On submit they also run on queries you have not seen, including one that returned no documents at all.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

on queries.json

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Recall@k, Precision@k and MRR for RAG retrieval, computed from document IDs

AI Testing for QA EngineersIntermediateTesting RAG and hallucination

How it works

Retrieval is tested separately from answers, using document IDs. For each query you know which documents are relevant and which ranked list the retriever returned. Recall@k is the share of relevant documents in the top k. Precision@k is the share of the top k slots that hold a relevant document, divided by k even when fewer results came back. Mean reciprocal rank rewards putting the first relevant document near the top.

Empty results are the edge case that breaks most implementations. A query that returns nothing scores zero on every metric, and the code must not divide by zero.

About the assignment

Implement the three metric functions and average them over the queries at k equals three, including queries with empty results.

What you will do

  1. Implement each metric for one query.
  2. Divide precision by k, not by the number returned.
  3. Return zero for a query with no relevant document returned.

Common mistakes

  • Dividing precision by the result count

    The page says to divide by k. Fewer results is worse, not neutral.

  • Crashing on an empty list

    Handle empty results before indexing or dividing.

In a real test suite

Teams track these metrics per release of a retriever, and per segment of questions, because a change that helps one kind of question can hurt another.

Keep the relevant-document lists versioned with the queries. When documents are renamed or merged, stale relevance labels make a good retriever look bad, which is the retrieval version of the stale gold problem from lesson 39.

Related lessons

Next lessonReal retrieval with real embeddingsAll lessons in the courseAI Testing for QA Engineers