Test runner basics (TestNG or pytest)

A test runner is the tool that finds your tests, runs set-up before them and clean-up after them, and records whether each one passed. Selenium only drives the browser; it has no idea what a test is. In Java the runner is usually TestNG or JUnit 5. Maven is a build tool: it downloads the libraries, compiles the code and starts the runner. Interviewers ask about runners early, because every other framework question builds on them: annotations and their order, assertions, and how you run tests from the command line.

A TestNG test class

An annotation is a tag that starts with @, such as @Test. It tells TestNG what a method is for.

public class LoginTest {
    private WebDriver driver;

    @BeforeMethod
    public void setUp() {
        driver = new ChromeDriver();
    }

    @Test
    public void validLoginShowsDashboard() {
        new LoginPage(driver).loginAs("asha", "secret");
        Assert.assertEquals(driver.getTitle(), "Dashboard");
    }

    @AfterMethod(alwaysRun = true)
    public void tearDown() {
        driver.quit();
    }
}

Every @Test method is one test. The set-up method runs before it, and the teardown method (the clean-up step) runs after it. alwaysRun = true makes the teardown run even when you select groups or when set-up failed, so no browser is left open.

The order of the annotations

TestNG has four levels, and they nest like boxes. A suite file (testng.xml) holds one or more <test> blocks, a <test> block holds classes, and a class holds test methods. "Before" methods open the boxes from the outside in, and "after" methods close them from the inside out.

A common set-up: load configuration and start the report in @BeforeSuite, open a fresh browser in @BeforeMethod, and take a screenshot on failure and quit the browser in @AfterMethod.

The classic trap is @BeforeTest. It does not mean "before each test method". It runs once for each <test> tag in the suite file. For "before every test method" you need @BeforeMethod.

Order of test methods

Without priorities or dependencies, TestNG 7 in practice runs the test methods of a class in alphabetical order of their names, not in the order they appear in the file. Its documentation does not promise any order for such methods, so never rely on it. priority sets an order: lower numbers run first, and a missing priority counts as 0. Good tests do not depend on order at all.

Hard and soft assertions

An assertion is a check that fails the test when it is not true. Assert.assertEquals(actual, expected, message) is a hard assertion: the test stops at the first failure. TestNG puts the actual value first, while JUnit puts the expected value first. Always add a message so the report explains the failure.

A soft assertion records failures and reports them together at the end:

SoftAssert soft = new SoftAssert();
soft.assertEquals(summary.total(), "120.00", "total");
soft.assertEquals(summary.tax(), "20.00", "tax");
soft.assertAll();   // fails here, listing both if both are wrong

If you forget assertAll(), the test passes even when checks failed. Create one SoftAssert in each test method.

Three more @Test attributes come up often. expectedExceptions passes a test only when it throws that exception. timeOut, in milliseconds, fails a test that runs too long. enabled = false switches a test off.

JUnit 5 in one view

Some teams use JUnit 5 (also called Jupiter) instead. @BeforeEach and @AfterEach run around every test. @BeforeAll and @AfterAll run once per class and must be static by default, because JUnit creates a new object for each test (a static method belongs to the class, not to one of those objects). @Disabled switches a test off, @Tag puts a label on tests so you can run them as a group, and @ParameterizedTest runs one test many times with different data. The default method order is fixed but deliberately not obvious; @TestMethodOrder with @Order sets it when you must.

TestNG is popular for UI suites because suite files, groups (labels such as "smoke" that let you run part of the tests), data providers (methods that feed data into a test), dependencies, retries and listeners (code that reacts to test events) are built in. Teams whose developers already use JUnit 5 often keep one runner for everything.

Maven and Surefire

pom.xml lists the libraries (Selenium, TestNG or JUnit, REST Assured), plugins and properties. Maven's main build phases run in this order: validate, compile, test, package, verify, install, deploy (many more, such as test-compile, sit between them). Running a phase runs all the earlier ones too.

mvn clean test                                       # fresh build, run all tests
mvn test -Dtest=LoginTest#validLoginShowsDashboard   # one method
mvn test -Dgroups=smoke                              # only tests labelled "smoke"
mvn test -Dbrowser=firefox                           # a value the tests can read

The Surefire plugin runs the tests in the test phase and writes results to target/surefire-reports, which CI servers read. A CI server (continuous integration) is a machine that runs your tests automatically, for example after every code change. Any -D value reaches the tests as a system property, a named setting the Java code can read with System.getProperty("browser"). Pointing Surefire at a suite file with suiteXmlFiles lets one command run testng.xml.

How to answer in an interview

For "explain the TestNG annotations", give the order first, then one real use for each level, then the @BeforeTest trap. For example: "Suite, test, class, method on the way in, and the reverse on the way out. I load config in @BeforeSuite, start a browser in @BeforeMethod and quit it in @AfterMethod with alwaysRun = true." Then stop and let the interviewer ask more.

Common mistakes

• Saying @BeforeTest runs before every test method.
• Forgetting assertAll() on a SoftAssert, or sharing one between tests.
• Writing tests that only pass in one order.
• Mixing up mvn test and mvn install: install also packages the project and copies it to the local repository.

Recap

The runner finds and runs tests; Selenium drives the browser. TestNG set-up goes suite, test, class, method, and teardown goes back out. Hard assertions stop a test, while soft ones report everything at assertAll(). Maven's Surefire plugin runs the tests, and -D values choose what runs and how.