Skip to content
CodeKruPro

Score an extractor

69/99

Module · Testing agents, chat UIs and extractors

advanced

about 40 min

What this lesson teaches

The idea

Extraction features turn documents into fields, such as an invoice into vendor, date and amount. One accuracy figure hides which fields are broken.

Why a tester cares

Scoring each field separately shows what to fix first, and how far each filled field can be trusted.

Words you will meet

Extractor:
a feature that pulls named fields out of a document.
Precision:
how often a filled field is right.
Recall:
how much of the real data was captured.
F1:
one number that balances precision and recall.

What you will do

  1. 1Score each field
  2. 2Build the confusion matrix
  3. 3Read which fields are weak
  4. 4Submit
Jump to practice

Extraction features turn documents into fields: an invoice into vendor, number, date, amount, tax ID and category. One accuracy number hides which fields are broken. Score each field separately:

  • True positive: the prediction is filled and equals the gold value.
  • False positive: the prediction is filled and wrong, or filled when the gold is empty.
  • False negative: the gold is filled and the prediction is empty or wrong.

A wrong value is both a false positive and a false negative. Precision says how often a filled field can be trusted; recall says how much of the real data was captured; F1 balances the two.

For a field with fixed values, such as the expense category, a confusion matrix shows which values get mixed up.

Your task

Score 60 labelled invoices. On submit your code also runs on 40 invoices you have not seen, with more empty fields.

Sign in to practise

The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.

Sign inCreate a free account
1

Your code

on invoices.json

How this is checked: When you submit, your script runs on 2 sets of data, 1 of them hidden, and the server compares what it prints with the right values. Run tries it on the data you can see.

2

Question

Which field should the team fix first?

How this is checked: One answer is right.

Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.

About this lesson

Evaluate an AI document extractor field by field with precision, recall, F1 and a confusion matrix

AI Testing for QA EngineersAdvancedTesting agents, chat UIs and extractors

How it works

Extraction features turn documents into fields, such as an invoice into vendor, number, date, amount, tax ID and category. One accuracy figure hides which fields are broken, so each field is scored separately.

A filled prediction equal to the gold value is a true positive. A filled prediction that is wrong, or filled when the gold is empty, is a false positive. A gold value the prediction missed or got wrong is a false negative. A wrong value is both. Precision says how far a filled field can be trusted; recall says how much of the real data was captured; F1 balances them. For a field with fixed values, a confusion matrix shows which values get mixed up.

About the assignment

Score labelled invoices field by field, build the category confusion matrix, and say which field the team should fix first.

What you will do

  1. Treat empty strings as no value, not as a value to compare.
  2. Count true positives, false positives and false negatives per field.
  3. Build the matrix with every category pair present, including zeros.

Common mistakes

  • Counting two empties as a match

    An empty prediction for an empty gold is not a true positive. It is simply not counted.

  • Dividing by zero

    F1 is zero when precision and recall are both zero.

In a real test suite

Per-field scores let teams set different thresholds per field: an amount must be nearly perfect, while a free-text description can tolerate more error.

Related lessons

Next lessonFour questions for any AI featureAll lessons in the courseAI Testing for QA Engineers