Skip to content
CodeKruPro

Is every claim in the context?

57/99

Module · Testing RAG and hallucination

intermediate

about 45 min

What this lesson teaches

The idea

An answer with citations looks trustworthy. Grounding tests check the trust is earned: the cited document exists, and it really says what the answer claims.

Why a tester cares

The sneakiest failure is an extra claim placed next to a correct citation. A simple word-overlap rule catches it without an AI judge.

Words you will meet

Citation:
a reference to the document an answer is based on.
Grounded:
supported by the cited text.
Unsupported claim:
a statement the cited document does not make.

What you will do

  1. 1Check citations exist
  2. 2Check numbers
  3. 3Check sentence support
  4. 4Submit
Jump to practice

A RAG answer with citations looks trustworthy. Grounding tests check that the trust is earned:

  1. Every cited document exists and opens.
  2. The facts in the answer, such as its numbers, appear in the cited document.
  3. Every factual sentence is supported by the cited text. The sneakiest failure is an extra claim placed beside a correct citation.

The third check needs a rule you can apply without a judge. A simple one works well: most of a sentence's content words must appear in the cited document.

Your task

Write grounding tests for PolicyPal. On submit they run against correct builds and builds with wrong citations, invented sources and unsupported claims.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
PolicyPal APIShow
PolicyPal's API, at the address in CK_APP_URL:
  POST /policy/ask {"question": "...", "department": "sales"} -> {"answer", "citations": [ids], "retrieved", "trace_id"}
  GET  /policy/docs/<id> -> the document ({"id", "title", "text"}), 404 if it does not exist
  GET  /policy/search?q=...&k=3 -> ranked documents
1

Your code

Write it in
Both are graded the same way.
against PolicyPal

How this is checked: When you submit, your tests run against 5 versions of PolicyPal: 2 correct and 3 with a planted fault. Good tests pass on every correct version and fail on every faulty one. Run tries them on the practice copy first; the result table shows each version.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

RAG grounding test: citations that exist, numbers that match and claims the source supports

AI Testing for QA EngineersIntermediateTesting RAG and hallucination

How it works

An answer with citations looks trustworthy, and grounding tests check that the trust is earned. Every cited document should exist and open. The facts in the answer, such as numbers, should appear in the cited document. Every factual sentence should be supported by the cited text.

The sneakiest failure is an extra claim placed beside a correct citation. A rule that needs no judge works well for it: most of a sentence's content words must appear in the cited document.

About the assignment

Write grounding tests for PolicyPal that pass on correct builds and fail on builds with wrong citations, invented sources and unsupported claims.

What you will do

  1. Fetch each cited document and fail if it does not open.
  2. Check the answer's numbers against the cited text.
  3. Check each sentence's content words against the cited text.

Common mistakes

  • Exact sentence matching

    Correct answers paraphrase. Compare content words, not whole sentences.

  • Checking only the first sentence

    Unsupported claims often come last.

In a real test suite

Grounding checks run cheaply on every answer, so teams use them as a first filter and send only the doubtful answers to a judge or a person.

Set the word-overlap threshold by looking at correct and incorrect sentences side by side. Too high and paraphrased correct answers fail; too low and invented claims that reuse a few document words slip through.

Keep a few hand-picked tricky answers as fixtures for the grounding checks themselves, such as a correct paraphrase and an unsupported sentence that reuses document words, so a change to the checks is tested before it reaches the suite.

Related lessons

Next lessonHallucination without a corpusAll lessons in the courseAI Testing for QA Engineers