[SOLVED] Inconsistent Mockito errors between individual and full tests execution

Inconsistent Mockito errors between individual and full tests execution

I have a project written in Java 11 with thousands of test, some of them using the Mockito library. When running all the tests with mvn test, I get the following:

[ERROR] Tests run: 26987, Failures: 0, Errors: 6, Skipped: 0

This indicates 6 errors in the Unit Tests. However, I am experiencing two weird behaviours:

When removing the 6 tests, other 6 errors appear
When running the single test classes with -Dtest, they all seem to be working

All of the errors appearing are always InvalidUseOfMatchers or UnfinishedStubbing.

I suppose there is some kind of leakage or conflict, or that the order these tests are run matter. However, my goal is not to solve the errors, but rather to get consistent results upon the different ways of compiling. I need to get the errors also with -Dtest, or to get success when running them all. The same version of surefire seems to be used whith the different test execution modes.

All of the tests are part of public classes with no annotations, and seem to be made as follows:

@Test
public void checkEndDate_validatesEndDateIsSet() {
    // Create a mock object for Timestamp
    Timestamp mockEndDate = Mockito.mock(Timestamp.class);

    // Create an instance of PerformancePeriodVO
    PerformancePeriodVO performancePeriod = new PerformancePeriodVO();

    // Assign the mocked Timestamp object
    performancePeriod.setEndDate(mockEndDate);

    // Ensure that getEndDate returns a non-null value
    Assertions.assertNotNull(performancePeriod.getEndDate(), "End date should not be null");
}

Hence, an instance of the class is instantiated inside the test, and each test hash the annotation @Test before it. The method shown is one of those returning the error InvalidUseOfMatchers.

The POM does contain the following:

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

It is the only thing relevant to unit tests I was able to find, but I can share more about it if needed.

As you might have noticed, this project was not written by me, nor am I an experienced Java developer, so I hope the question is exhaustive enough

UPDATE

As asked, here is a full class. In this case, the method failing is getHash_testGetHashSameInstance at its first line. However, when run with -Dtest, I get no errors at all.

package it.dap.nw.commons.db;

import java.lang.reflect.Field;
import java.util.Hashtable;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.Assertions;
import org.mockito.Mockito;

public class PropertyOverrideTest {
    @Test
    public void getHash_testGetHashNotNull() {
        Hashtable hashMock = Mockito.mock(Hashtable.class);
        PropertyOverride.setHash(hashMock);
        Hashtable result = PropertyOverride.getHash();
        Assertions.assertNotNull(result);
    }


    @Test
    public void getHash_testGetHashSameInstance() {
        Hashtable hashMock = Mockito.mock(Hashtable.class);
        try {
            Field field = PropertyOverride.class.getDeclaredField("hash");
            field.setAccessible(true);
            field.set(null, hashMock);
        } catch (NoSuchFieldException | IllegalAccessException e) {
            e.printStackTrace();
        }
        Hashtable result = PropertyOverride.getHash();
        Assertions.assertEquals(hashMock, result, "Expected same instance of Hashtable");
    }


    @Test
    public void setHash_testSetHashNotNull() {
        Hashtable hashMock = Mockito.mock(Hashtable.class);
        PropertyOverride.setHash(hashMock);
        Assertions.assertEquals(hashMock, PropertyOverride.getHash());
    }


    @Test
    public void setHash_testSetHashNull() {
        Hashtable hashMock = Mockito.mock(Hashtable.class);
        PropertyOverride.setHash(null);
        Assertions.assertNull(PropertyOverride.getHash());
    }
}

Solution

Surefire was incorrectly marking many tests as failed, including those provided as examples, even though they were actually functioning correctly. This explains why running the tests with -Dtest revealed no errors.

Upon closer inspection of the Mockito logs located just below the error messages in Surefire, I discovered that the issue was related to entirely different classes. The problem was ultimately resolved by adding @ExtendWith(MockitoExtension.class) before the class declarations.