Skip to content
CodeKruPro

Real retrieval with real embeddings

56/99

Module · Testing RAG and hallucination

intermediate

about 45 min

What this lesson teaches

The idea

How documents are cut into pieces and turned into numbers decides what retrieval can find, before any answer is written.

Why a tester cares

Measuring two ways of cutting the same documents on the same questions shows which one works, instead of guessing.

Words you will meet

Chunk:
a piece of a document that is searched on its own.
Embedding:
a list of numbers that captures what a piece of text means, so similar texts get similar numbers.
Hit rate:
how often the right chunk comes back.

What you will do

  1. 1Chunk by section
  2. 2Chunk by size
  3. 3Compare hit rates
  4. 4Choose one
Jump to practice

Retrieval quality depends on choices made before any model answers: how documents are cut into chunks, and how chunks are embedded. Here you use a real sentence-embedding model, all-MiniLM-L6-v2, running in the lesson's runner through the fastembed library.

Two common chunking choices:

  • By section: one chunk per heading. Keeps context together, but a long section that covers many things gets one blurred embedding.
  • Fixed size: chunks of about 200 characters. Each is about one thing, but loses its heading.

Your task

  1. Implement both chunking functions.
  2. Run them: the script embeds the chunks and reports, for 12 paraphrased questions, how often the single best chunk comes from the right section.
  3. Decide which you would ship for this handbook. On submit your code also runs on a second handbook.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

with the embedding model

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

2

Question

Which chunking would you ship for this handbook?

How this is checked: One answer is right.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Chunking strategy and embeddings for RAG retrieval, compared with a real embedding model

AI Testing for QA EngineersIntermediateTesting RAG and hallucination

How it works

Retrieval quality depends on choices made before any answer: how documents are cut into chunks, and how chunks are embedded. This lesson uses a real sentence-embedding model, all-MiniLM-L6-v2, through the fastembed library, running inside the lesson's runner with no internet.

Chunking by section keeps context together, but a long section that covers many things gets one blurred embedding. Fixed-size chunks are each about one thing, but lose their heading. Measuring both on the same paraphrased questions shows which works for a given corpus.

About the assignment

Implement section and fixed-size chunking, compare their hit rates on paraphrased questions, and choose one.

What you will do

  1. Split by headings for the section chunker.
  2. Split into chunks of about the size the page gives for the fixed chunker.
  3. Run both and compare how often the right chunk comes first.

Common mistakes

  • Cutting words in half

    Fixed-size chunks that split mid-word embed worse. Break at spaces.

  • Choosing by intuition

    The point is to measure. A corpus can favour either choice.

In a real test suite

Real teams rerun this comparison whenever the corpus changes shape, for example when long policy documents arrive, and keep the evaluation questions fixed so results are comparable.

Measure on questions written in your users' words, not copied from the documents. Paraphrased questions are harder for retrieval, and they are what real users send, so they give a more honest comparison between chunking choices.

Record the model name and version with every result. Embedding models change too, and a hit rate measured with one model says little about another.

Related lessons

Next lessonIs every claim in the context?All lessons in the courseAI Testing for QA Engineers