"AI" in a job title tells you little. Three different jobs share the label, and a fourth kind of post just adds the word to ordinary QA work. Read the duties, not the title:
Tests AI products: golden sets, judges, red teaming, RAG metrics. This course prepares you for these.
Uses AI tools to test ordinary software: Copilot-written tests, self-healing locators. Useful, and closer to your current job.
ML engineering: training models and pipelines. A different career.
AI in the title only: the duties are ordinary QA.
The posts below are composites written for the course from patterns in real postings, so no company is named. Real postings change fast: practise the same reading on live ones.
Sign in to practise
The lesson is free to read. A free account lets you talk to the practice bots, run your code and save your score. You will come back to this lesson.
1AI QA Engineer, fintech chatbotBuild and maintain golden datasets for our support bot, run LLM-judge evals in CI, measure judge agreement with human labels, and report regressions by intent before each release.
2QA Engineer (AI)Write and maintain Selenium regression tests for our web portal, log defects in Jira, take part in sprint ceremonies. Knowledge of AI is a plus.
3SDET, GenAI productivityUse GitHub Copilot and Claude to speed up API test automation for our payments platform; review generated tests and keep coverage above 80%.
4Machine Learning Engineer, QualityTrain and tune ranking models, own feature pipelines in Spark, run offline experiments and ship model improvements.
5LLM Evaluation SpecialistDesign rubrics with domain experts, label conversations, calibrate an LLM judge against human labels, and track pass rates with confidence intervals.
6AI Test Automation EngineerMaintain our Cypress suite and roll out an AI self-healing locator tool across 3 product teams. Manual exploratory testing as needed.
7Senior QA, AI Platform teamManual and regression testing of our HR SaaS web app, test case writing in TestRail, release sign-off. You will sit in the AI Platform team.
8Trust and Safety Test EngineerRun prompt-injection and jailbreak suites against our assistant, turn findings into regression tests, and test data-leak paths through retrieved documents.
9Applied Scientist, SearchTrain embedding models for product search, run A/B tests on ranking, publish internal papers on retrieval quality.
10QA Lead, AI TransformationLead a team of 8 manual testers and introduce AI tools that draft test cases from user stories, cutting test design time by half.
11AI QA AnalystTest mobile app UI flows on Android and iOS, write test cases, report bugs. Some exposure to AI-based features is desirable.
12RAG Quality EngineerMeasure retrieval recall@k and MRR, check groundedness and citation accuracy of answers, and build regression sets from production traces.
How this is checked: Each row has a right label. At least 9 of 12 must match. For the full score, a harder check: a few key rows must all be right. All 12 must match.
Core checks earn 50 points and the harder checks the other 50. You can submit as many times as you like.
About this lesson
How to read an AI QA job description: duties, not titles, tell you what the job is
AI Testing for QA Engineers·Advanced·Career, capstones and the Cold Product Test
How it works
The word AI in a job title says little. Several different jobs share the label, and some posts add it to ordinary QA work. Reading the duties shows which job it is.
Posts that test AI products talk about golden sets, judges, red teaming and retrieval metrics. Posts that use AI tools to test ordinary software talk about assistants writing tests and self-healing locators. Machine learning engineering posts talk about training models and pipelines. And title-only posts list duties with no AI at all.
About the assignment
Classify twelve composite job posts by their duties into the four kinds.
What you will do
1Read the duties, not the title.
2Underline the verbs: build, train, measure, write, use.
3Mark a post title-only when no duty involves AI at all.
Common mistakes
Trusting the title
Several posts with AI in the title describe ordinary QA.
Confusing tool use with AI testing
Using an assistant to write tests is valuable, but it is not testing an AI product.
In a real test suite
Practise the same reading on live postings in your city or remote market. Count how many are real AI testing roles: the number is usually smaller than the search results suggest, and knowing it helps you target applications.
In interviews for title-only roles, asking how the team tests its AI features often reveals whether there is any AI work at all.
