Skip to content
CodeKruPro

Practice

AI Testing for QA Engineers

Learn to test AI products with 99 hands-on assignments, from your first prompt-injection test to a full release gate. Write real evals in Python, catch wrong answers, leaks and unsafe agent actions, and get graded against practice bots. No API key or GPU needed.

  • 99 lessons
  • 16 modules
  • Beginner to Advanced
  • Lessons 1 to 12 free

Keep your progress

Signed-in learners keep solved lessons and a leaderboard rank. Lessons also work without an account.

Sign in to save progress

Why exact match fails on AI

7 lessonsBeginner
  1. Same question, different answers
  2. Same question, real model
  3. Your first pass or fail rule
  4. Test a refund chatbot whose answers change every time
  5. One run is not evidence
  6. Make the bot leak a secret
  7. Which bugs are not AI bugs

Python and LLM APIs bridge

3 lessonsBeginner
  1. Python for testers: JSON and safe parsing
  2. Call an LLM API with the OpenAI SDK
  3. pytest parametrize, fixtures and markers for evals

A deterministic harness

7 lessonsBeginner
  1. Schema tests for structured output
  2. Property checks: facts, forbidden content, length
  3. API tests around the model
  4. Metamorphic tests when no oracle exists
  5. Build a tiny eval harness
  6. Test the test: false alarms and misses
  7. Your harness against a real model

Read before you measure

6 lessonsBeginner to Intermediate
  1. Read a trace like a bug report
  2. Note what went wrong, and where it first went wrong
  3. Turn notes into a failure taxonomy
  4. Code, judge, human, or not a test
  5. Who decides what correct means
  6. A one-page test approach for an AI feature

Working with an AI assistant as a tester

4 lessonsIntermediate
  1. Give the assistant what a tester knows
  2. Instruction files for test generation
  3. Generate, then prove it
  4. Use the assistant for the boring parts

Reviewing AI-written tests

5 lessonsIntermediate
  1. Why a green AI-written suite can miss everything
  2. Fix URL-only and visible-only assertions
  3. Tests that mirror the bug
  4. Mutation testing as review
  5. Heal or reject

Agents in the browser: Playwright MCP and test agents

5 lessonsIntermediate
  1. What the agent sees
  2. Be the agent
  3. MCP or CLI: how an agent should drive the browser
  4. Planner, generator, healer: what each may change
  5. Review an agent's run

Datasets

3 lessonsIntermediate
  1. Build a 30-case golden set, and name its purpose
  2. Audit the gold
  3. When the set stops representing traffic

Statistics for testers

6 lessonsIntermediate
  1. A pass rate is a range
  2. Find your noise floor
  3. Noise or regression
  4. Same cases, paired comparison
  5. How many cases do you need?
  6. pass@k, pass^k and severity overrides

Judges and guardrails as classifiers

7 lessonsIntermediate
  1. Write a binary rubric
  2. A judge is a classifier: TPR and TNR
  3. The accuracy trap at a 5% failure rate
  4. Cohen's kappa and the human ceiling
  5. Train, dev and test splits for judges
  6. Correct an estimate, catch judge drift
  7. A guardrail is a classifier

Testing RAG and hallucination

8 lessonsIntermediate to Advanced
  1. Separate retrieval from generation
  2. Recall@k, Precision@k and MRR by hand
  3. Real retrieval with real embeddings
  4. Is every claim in the context?
  5. Hallucination without a corpus
  6. When the answer is not in the documents
  7. Debug PolicyPal
  8. Real failure hunt

Testing agents, chat UIs and extractors

8 lessonsAdvanced
  1. Grade the outcome, not the claim
  2. Tool choice and arguments
  3. Faults, loops and budgets
  4. When the path is a requirement
  5. Multi-turn: first failure and N-1 replay
  6. MCP contract test: pin and diff tools
  7. Test the chat window
  8. Score an extractor

Security, safety and evidence

9 lessonsAdvanced
  1. Four questions for any AI feature
  2. Indirect injection through a document
  3. Exfiltration through rendered output
  4. Excessive agency: hold the limit at the API layer
  5. PII in output and logs
  6. Scanner triage with a curated garak run
  7. Review a system prompt as a spec
  8. Unseen attacks, then a regression test
  9. Bias checks and an evidence table

Release, production and operations

10 lessonsAdvanced
  1. A prompt change is a deploy
  2. Build a gate script
  3. CI tiers with a cost budget
  4. A model upgrade: what blocks it
  5. Latency, throughput and token cost under load
  6. Read a LangSmith export
  7. Failure rate by segment with SQL
  8. Production sampling
  9. Ship or hold with conflicting evidence
  10. Defend the plan

Frameworks and other languages

3 lessonsAdvanced
  1. Re-run your harness in DeepEval
  2. Promptfoo assertions, a CI gate, and moving an eval
  3. Port your harness to Java or JavaScript

Career, capstones and the Cold Product Test

8 lessonsAdvanced
  1. Read a job post for evidence
  2. Benchmarks and how they get gamed
  3. Rewrite your CV for AI testing
  4. Answer the ten questions that always come
  5. Capstone 1: PolicyPal evaluation report
  6. Capstone 2: OrderDesk release gate
  7. Capstone 3: generate, then review, an AI-written suite
  8. Cold Product Test

About this course

AI Testing for QA Engineers

What is AI testing?

AI testing means testing products built on large language models (LLMs): support chatbots, search that answers from documents (RAG), and agents that take actions such as refunds. Their answers change from one run to the next, so the exact-match assertions QA engineers rely on break on correct replies and still miss real faults.

Testing them well means checking the facts and properties every correct reply must have, measuring pass rates with evals instead of trusting one run, checking LLM judges against human labels, and probing for prompt injection, data leaks and unsafe agent actions.

What does an AI QA engineer do?

An AI QA engineer tests the AI features of a product, with the same aim as any tester: find what is broken before customers do, and give the team evidence it can act on. The work looks like this:

  • Build golden sets and eval harnesses that run on every change, and report failures by kind.
  • Read traces to find the first step that went wrong: understanding, retrieval, a tool call or the answer.
  • Check that an LLM judge agrees with people before trusting its numbers, and tell a real regression from noise.
  • Test RAG retrieval and grounding, agent tool calls and outcomes, chat windows and extractors.
  • Red-team for prompt injection, data leaks and excessive agency, and record the evidence.
  • Run release gates in CI, and review tests that AI coding assistants write.

Each of these has graded lessons in this course.

Who this course is for

  • Manual testers who want to move into AI testing and learn just enough Python on the way.
  • Automation engineers and SDETs (Selenium, Playwright, REST Assured) who have been asked to test an AI feature.
  • QA leads who need a test approach, release gates and a budget for an AI product.

You need testing basics. Python basics are taught in lessons 8 to 10. You do not need machine learning, maths beyond school level, an API key or a paid AI model.

What you will learn: 16 modules

  1. 01Why exact match fails on AI. Non-deterministic replies, repeats, prompt injection and ordinary software faults.
  2. 02Python and LLM APIs. JSON parsing, the OpenAI Python SDK with timeouts and retries, pytest parametrize and fixtures.
  3. 03A deterministic harness. Schema, property, API and metamorphic tests, a small eval harness, and running it on a real model.
  4. 04Read before you measure. Traces, error analysis, failure taxonomies and a one-page test approach.
  5. 05Working with an AI assistant. Prompts and AGENTS.md files that make generated tests catch bugs.
  6. 06Reviewing AI-written tests. Escapes, weak assertions, mirror tests, mutation testing and self-healing diffs.
  7. 07Playwright MCP and test agents. What a browser agent sees, MCP or CLI, and reviewing an agent's run.
  8. 08Datasets. Golden sets, auditing stale answers and drift from real traffic.
  9. 09Statistics for testers. Pass-rate intervals, noise floors, regressions, sample size, pass@k and pass^k.
  10. 10Judges and guardrails. Binary rubrics, TPR and TNR, precision and recall, Cohen's kappa and judge drift.
  11. 11Testing RAG and hallucination. Retrieval metrics, embeddings, grounding, abstention and a real-model failure hunt.
  12. 12Testing agents, chat UIs and extractors. Outcomes, tool calls, loops, multi-turn replays, MCP contract tests and field scoring.
  13. 13Security, safety and evidence. Indirect injection, exfiltration, excessive agency, PII masking, scanner triage and bias checks.
  14. 14Release and operations. Run manifests, gate scripts, CI tiers, model upgrades, load tests, traces and production sampling.
  15. 15Frameworks and other languages. DeepEval, Promptfoo, and porting the harness to Java or JavaScript.
  16. 16Career and capstones. Job posts, CV, interview questions, three capstones and the Cold Product Test.

How the course works

  • Each lesson gives you a practice product with faults planted in it: a refund chatbot, an HR policy bot that answers from documents, an order agent that issues refunds, an online store and more.
  • You write tests, answers or reports in the browser. On submit they are graded against hidden correct and faulty versions of the product: core checks earn 50 points and harder checks the other 50.
  • Every lesson starts with a plain-language summary of the idea and says how each answer is checked. Written answers come with a model answer to compare once you submit.
  • Lessons 1 to 12 are free with a free account. CodeKru Pro opens the rest and adds a course certificate when you finish.

Questions people ask

Is this AI testing course free?

Lessons 1 to 12 are free with a free CodeKru account: the first two modules and the start of the eval harness module. CodeKru Pro opens all 99 lessons and the course certificate. It is one payment and nothing renews.

Do I need to know machine learning or Python?

No machine learning is needed. The course teaches the Python you use (JSON, the OpenAI SDK and pytest) in lessons 8 to 10, and later lessons explain each statistic before you use it. If you work in Java or JavaScript, lesson 91 ports the eval harness to JUnit with REST Assured or to Playwright Test.

Do I need an API key or a paid AI model?

No. The practice bots and your code both run on CodeKru's servers; you only need a browser. A few lessons compare the practice bot with a real model through CodeKru, at no cost to you.

What tools does the course cover?

Python with pytest, the OpenAI Python SDK, JSON Schema, DeepEval, Promptfoo, Playwright, Playwright MCP, SQL, GitHub Actions, trace exports from LangSmith and Langfuse, a curated garak scan, and JUnit with REST Assured.

Is this about testing AI, or using AI to test?

Mostly testing AI products: chatbots, RAG and agents. Two modules also cover using AI coding assistants to write tests and reviewing the tests they write, because most testers now do both.

How long does the course take?

About 70 hours of hands-on work across 99 lessons. Most lessons take 15 to 60 minutes; the capstones and the Cold Product Test take longer. You can stop and come back at any point, and your work is saved.

Will this help me prepare for AI QA engineer interviews?

The last module covers reading AI testing job posts, writing CV lines with evidence, the questions interviewers keep asking, three capstone projects and a take-home style Cold Product Test. It is practice evidence for interviews, not a job guarantee.