selenium-webdriver testng gui-testing testng.xml

How do I pass the correct driver of a failed (java) selenium test to a testng ITestListener, when running multiple tests?

I have a TestListener class, which implements ITestListener, which is responsible for taking pictures and saving the html if a selenium test fails.

My selenium tests have a parent class BaseTest, which helps do common tasks, like initiating each driver.

The content of each important area:

TestListener

public class TestListener implements ITestListener {
    WebDriver driver=null;
    ITestContext context = null;
    String filePath = "artifacts/";

    @Override
    public void onTestFailure(ITestResult result) {
        ITestContext context = result.getTestContext();
        driver = (WebDriver) context.getAttribute("WebDriver");
        System.out.println("***** Error "+result.getName()+" test has failed *****");
        String methodName=result.getName().toString().trim();
        String currentTime = getCurrentTime();
        saveScreenShot(methodName, currentTime);
        savePageSource(methodName, currentTime);
        saveConsoleLog(methodName, currentTime);
    }
...

BaseTest (each test calls this separately)

    private void testngSetup(ITestContext context){
        context.setAttribute("WebDriver", driver);
        TestRunner runner = (TestRunner) context;
        runner.setOutputDirectory(artifactLocation);
    }

pom.xml

        <plugin>
          <artifactId>maven-surefire-plugin</artifactId>
          <version>3.2.5</version>
          <configuration>
            <suiteXmlFiles>
              <suiteXmlFile>testng.xml</suiteXmlFile>
            </suiteXmlFiles>
            <useSystemClassLoader>false</useSystemClassLoader>
          </configuration>
        </plugin>

testng.xml

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="esm4hpc">
    <listeners>
        <listener class-name="selenium.listeners.TestListener"/>
    </listeners>
    <test name="sample-Test" verbose="2" parallel = "classes">
        <packages>
            <package name=".*"/>
        </packages>
    </test>
</suite>

I have found this thread (How to get the current class driver in ItestListener), which suggested to change getting the driver in ITestListener the following way: (WebDriver)result.getTestClass().getRealClass().getDeclaredField("driver").get(result.getInstance()), but it throw java.lang.NoSuchFieldException : driver

public class TestListener implements ITestListener {
    WebDriver driver=null;
    ITestContext context = null;
    String filePath = "artifacts/";

    @Override
    public void onTestFailure(ITestResult result) {
        ITestContext context = result.getTestContext();
        driver = (WebDriver)result.getTestClass().getRealClass().getDeclaredField("driver").get(result.getInstance());
        System.out.println("***** Error "+result.getName()+" test has failed *****");
        String methodName=result.getName().toString().trim();
        String currentTime = getCurrentTime();
        saveScreenShot(methodName, currentTime);
        savePageSource(methodName, currentTime);
        saveConsoleLog(methodName, currentTime);
    }
...

I have also tried to set the parallel tag in testng.xml to false, but did not seem to help. I also tried to list each testing class separately, but it also gave back wrong pictures.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="esm4hpc">
    <listeners>
        <listener class-name="selenium.listeners.TestListener"/>
    </listeners>
    <test name="sample-Test" verbose="2" parallel = "false">
        <classes>
            <class name="selenium.esm4hpc.AuthenticationTest"/>
            <class name="selenium.esm4hpc.FileEditorTest"/>
            <class name="selenium.esm4hpc.FileManagerTest"/>
            <class name="selenium.esm4hpc.JobTest"/>
            <class name="selenium.esm4hpc.ProjectTest"/>
        </classes>
    </test>
</suite>

The only thing seemed to help is to move each test class into different test-cases:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="esm4hpc">
    <listeners>
        <listener class-name="selenium.listeners.TestListener"/>
    </listeners>
    <test name="AuthenticationTest" verbose="2" parallel = "false">
        <classes>
            <class name="selenium.esm4hpc.AuthenticationTest"/>
        </classes>
    </test>
    <test name="FileEditorTest" verbose="2" parallel = "false">
        <classes>
            <class name="selenium.esm4hpc.FileEditorTest"/>
        </classes>
    </test>
    <test name="FileManagerTest" verbose="2" parallel = "false">
        <classes>
            <class name="selenium.esm4hpc.FileManagerTest"/>
        </classes>
    </test>
    <test name="JobTest" verbose="2" parallel = "false">
        <classes>
            <class name="selenium.esm4hpc.JobTest"/>
        </classes>
    </test>
    <test name="ProjectTest" verbose="2" parallel = "false">
        <classes>
            <class name="selenium.esm4hpc.ProjectTest"/>
        </classes>
    </test>
</suite>

But this causes the result files to be generated separately, which makes it cumbersome to debug failed cases.

Solution

Let's start with clearing out a few details.

ITestContext is TestNG's API way of pointing to a <test> tag (from the suite lingo)
ITestResult is TestNG's API way of pointing to the result of the execution of a particular @Test or a configuration method.
There can be one or more test methods that belong to the same ITestContext. So one ITestContext can have one or more ITestResult objects in it.
From within a @Test method, if you invoke org.testng.Reporter.getCurrentTestResult() you will always have access to the currently running test method's result.
ITestResult.setAttribute() method lets you attach attributes to a particular test method, similar to how ITestContext.setAttribute() allows you to attach attributes at the <test> tag level and ISuite.setAttribute() attaches attributes at the <suite> tag level.

To fix issues that you are experiencing you would need to do the following:

In your test listener, change the line driver = (WebDriver) context.getAttribute("WebDriver"); to driver = (WebDriver) testResult.getAttribute("WebDriver");
Modify your base class method which is responsible for instantiating a webdriver and have it set it to the ITestResult object of the currently running test, as an attribute using its setAttribute() method. You would need to ensure that the setup method is explicitly called via the @Test method or implicitly called via its run() method (If your base class is implementing IHookable interface that TestNG provides)

The easiest way of ensuring that there's absolute thread-safety when it comes to sharing WebDriver instances and also in getting access to it in both your test classes as well as in your listeners is to do something like shown in the below sample.

import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.ITestListener;
import org.testng.ITestResult;
import org.testng.Reporter;

import java.util.Optional;

public class WebDriverLifeCycleManager implements ITestListener {

    public static final String DRIVER = "driver";

    public static RemoteWebDriver driver() {
        return Optional.ofNullable(Reporter.getCurrentTestResult().getAttribute(DRIVER))
                .map(it -> (RemoteWebDriver) it)
                .orElseThrow(() -> new IllegalArgumentException("Could not find driver"));
    }

    @Override
    public void onTestStart(ITestResult result) {
        ChromeDriver driver = new ChromeDriver();
        result.setAttribute(DRIVER, driver);
    }

    @Override
    public void onTestSuccess(ITestResult result) {
        cleanupDriverReference(result);
    }

    @Override
    public void onTestFailure(ITestResult result) {
        //Add logic to capture screenshot etc.,
        cleanupDriverReference(result);
    }

    @Override
    public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
        cleanupDriverReference(result);
    }

    @Override
    public void onTestSkipped(ITestResult result) {
        cleanupDriverReference(result);
    }

    private void cleanupDriverReference(ITestResult result) {
        Optional.ofNullable(result.getAttribute(DRIVER))
                .map(it -> (RemoteWebDriver) it)
                .ifPresent(RemoteWebDriver::quit);
        result.removeAttribute(DRIVER);
    }
}

Here's a test class that explicitly uses the above listener (You can have this changed to use the service loading approach of automatically wiring in listeners as explained in the documentation )

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(WebDriverLifeCycleManager.class)
public class ThreadSafeWebDriverSample {

    @Test
    public void test() {
        WebDriverLifeCycleManager.driver().get("https://www.google.com");
    }
}