Groups, fixtures, listeners and retries

A small suite needs only @Test and a few set-up methods. A large UI suite needs more: a way to run only some tests, a way to feed data, code that reacts when a test fails, and a way to rerun failures. TestNG has a feature for each, and interviewers ask about all of them because each one hides a trap. This page goes through them in the order you would add them to a real framework.

Groups and the suite file

A group is a label on a test. One test can belong to several groups:

@Test(groups = {"smoke", "regression"})
public void login() { ... }

The suite file, testng.xml, then chooses what runs and how:

<suite name="Smoke" parallel="methods" thread-count="4">
  <listeners><listener class-name="com.acme.FailureListener"/></listeners>
  <parameter name="browser" value="chrome"/>
  <test name="smoke">
    <groups><run><include name="smoke"/><exclude name="broken"/></run></groups>
    <packages><package name="com.acme.tests"/></packages>
  </test>
</suite>

It lists classes or packages, groups to include or exclude, parameters, listeners and the parallel mode. (parallel and thread-count tell TestNG to run several tests at the same time; see Parallel and CI runs.) From Maven, -Dgroups=smoke does the group part. When you select groups, give set-up and teardown methods alwaysRun = true; otherwise they are filtered out too, and tests run with no browser.

DataProvider and Parameters

A @DataProvider supplies many rows of test data, and the test runs once per row:

@DataProvider(name = "logins")
public Object[][] logins() {
    return new Object[][] { {"admin", "a1", true}, {"blocked", "b1", false} };
}

@Test(dataProvider = "logins")
public void login(String user, String pass, boolean ok) { ... }

It may return Object[][] (a table: each inner array is one row) or Iterator<Object[]>, and the rows can come from CSV, JSON or Excel. @DataProvider(parallel = true) runs the rows in parallel, 10 at a time by default. A provider in another class needs dataProviderClass, and must be static (a method that belongs to the class itself) unless that class has a constructor with no arguments.

@Parameters("browser") reads one value from the suite file, and @Optional("chrome") gives a default. The rule of thumb: a DataProvider is for test data, and Parameters are for run settings.

Priority, dependencies and switching tests off

• priority: lower numbers run first, no priority counts as 0, and ties in practice run in method-name order.
• dependsOnMethods = "login": if login fails, the dependent test is skipped, not failed.
• enabled = false: the test does not run.
• invocationCount = 3: the method runs three times, which helps when you chase a flaky test (one that sometimes passes and sometimes fails with no code change).
• @Factory: a method that returns several instances of a test class, for example one per data set, so the same tests run with different constructor data.

Priority only orders tests; it does not create a dependency. Dependencies between UI tests make suites fragile, so prefer tests that set up their own state.

Sharing data without static fields

A test method can take an ITestContext parameter. Calling context.setAttribute("orderId", id) in one test and getAttribute("orderId") in a later one shares a value within the same <test> block. Use it sparingly, because tests that need each other cannot run in parallel safely.

Listeners

A listener is a class that TestNG calls when something happens in the run, so shared work stays out of the tests:

public class FailureListener implements ITestListener {
    @Override
    public void onTestFailure(ITestResult result) {
        // take a screenshot, attach it to the report, log the error
    }
}

ITestListener covers test start, success, failure and skip. ISuiteListener covers the start and end of the suite, IAnnotationTransformer changes annotations at run time, and IReporter builds custom reports. In TestNG 7 the listener methods have default bodies, so you write only the ones you need. Register a listener with @Listeners, in testng.xml, or through Java's service loader (a Java feature that finds classes named in a file).

Rerunning failed tests

When a run has failures, TestNG writes testng-failed.xml with the failed tests, the tests skipped because of them, and the set-up methods they need. Running that file repeats just those tests. For automatic retries, write a retry analyzer:

public class Retry implements IRetryAnalyzer {
    private int attempts = 0;
    public boolean retry(ITestResult result) {
        return attempts++ < 2;
    }
}

Attach it with @Test(retryAnalyzer = Retry.class), or to every test through an IAnnotationTransformer. Earlier attempts are reported as skipped, and the last attempt decides the result.

How to answer in an interview

Most questions here are "what is the difference between X and Y?" Answer with the purpose of each, then one line of code. For DataProvider versus Parameters: "A DataProvider feeds many rows and the test runs once per row; Parameters read one setting, such as the browser, from testng.xml." For subsets, say that you choose them with groups, suite files or -Dtest, never by editing test code.

Common mistakes

• Selecting groups and forgetting alwaysRun = true on set-up and teardown.
• Using priority as if it were a dependency.
• Retrying every test three times and calling the suite stable. Retries hide flaky tests; log each retry and fix the cause.
• Keeping shared values in static fields instead of letting each test create its own data.

Recap

Groups and suite files choose what runs. DataProviders feed test data, and Parameters feed settings. Priority sets order, while dependencies skip tests whose prerequisite failed. Listeners keep screenshots and logging out of tests, and testng-failed.xml or a retry analyzer reruns failures.