[SOLVED] Inconsistency when running java jar via CLI and IntelliJ

My project is in the following structure:

src
└── main
    └── java
        └── tests
            ├── functional
            │   ├── TestA
            │   └── TestB
            ├── verification
            │   └── TestC
            └── TestsRunner.java
target
├── classes
├── lib
│   ├── depA
│   ├── depB
│   └── ..
├── app.jar
└── ..

The main class, is at the TestsRunner.java class -

package tests;

import lombok.extern.slf4j.Slf4j;
import picocli.CommandLine;

import java.util.concurrent.Callable;


@Slf4j
@CommandLine.Command(name = "run", mixinStandardHelpOptions = true, version = "1.0", description = "Run tests")
public class TestsRunner implements Callable<Integer> {

  @CommandLine.Option(names = {"-s", "--suite"}, required = true, description = "Suite name")
  private String testSuiteName;

  @CommandLine.Option(names = {"-o", "--output"}, required = true, description = "Test reports output directory")
  private String reportOutputDir;

  public static void main(String[] args) {
    int exitCode = new CommandLine(new TestsRunner()).execute(args);
    System.exit(exitCode);
  }

  @Override
  public Integer call() throws Exception {
    var basePackage = switch (testSuiteName) {
      case "functional" -> "tests.functional";
      case "verification" -> "tests.verification";
      default -> throw new RuntimeException("Unknown tests suite: " + testSuiteName);
    };
    var runner = Junit5ServiceTestsRunner.builder()
      .classNamePattern(".*Test")
      .classNamePattern(".*Tests")
      .basePackage(basePackage)
      .reportOutputDir(reportOutputDir)
      .build();
    var result = runner.run();
    return result.isSuccess() ? 0 : 1;
  }
}

I am also using the maven-jar-plugin plugin in order to copy all dependencies to lib directory and state the main class

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-jar-plugin</artifactId>
    <version>3.2.0</version>
    <configuration>
        <finalName>app</finalName>
        <archive>
            <manifest>
                <addClasspath>true</addClasspath>
                <Main-Class>tests.TestsRunner</Main-Class>
                <classpathPrefix>lib</classpathPrefix>
                <!--workaround for https://issues.apache.org/jira/browse/MJAR-156-->
                <!--https://stackoverflow.com/questions/67505356/maven-jar-plugin-generates-jar-with-wrong-manifest-mf-->
                <useUniqueVersions>false</useUniqueVersions>
            </manifest>
            <manifestEntries>
                <Class-Path>.</Class-Path>
            </manifestEntries>
        </archive>
    </configuration>
</plugin>

When running the TestsRunner class via IntelliJ, I use the Modify run configurations to provide the -s and -o values. Running the application via IntelliJ works, the output contains the results of the tests that were executed (notice the 1 tests successful):

Test run finished after 4246 ms
[         3 containers found      ]
[         0 containers skipped    ]
[         3 containers started    ]
[         0 containers aborted    ]
[         3 containers successful ]
[         0 containers failed     ]
[         1 tests found           ]
[         0 tests skipped         ]
[         1 tests started         ]
[         0 tests aborted         ]
[         1 tests successful      ]
[         0 tests failed          ]

Now, I see that IntelliJ uses their own java agent, and it also states the classpath explicitly on each run:

"C:\Program Files\Java\jdk-17.0.8\bin\java.exe" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2022.1.3\lib\idea_rt.jar=65474:C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 2022.1.3\bin" -Dfile.encoding=UTF-8 -classpath C:\Users\xxx\Documents\GitHub\my-project\tests\target\classes;C:\Users\xxx\.m2\repository\info\picocli\picocli\4.7.1\picocli-4.7.1.jar;C:\Users\xxx\.m2\repository\com\core\testing\mock-servers-junit5\1.1.0\mock-servers-junit5-1.1.0.jar ..........(all other dependencies) tests.TestsRunner -s verification -o report

When running from the command line at the target directory using:

java -jar app.jar -o ${JUNIT_OUTPUT_FOLDER} -s ${SUITE_NAME}

It is as if it does not recognize or find any tests. I receive the following output:

Test run finished after 62 ms
[         1 containers found      ]
[         0 containers skipped    ]
[         1 containers started    ]
[         0 containers aborted    ]
[         1 containers successful ]
[         0 containers failed     ]
[         0 tests found           ]
[         0 tests skipped         ]
[         0 tests started         ]
[         0 tests aborted         ]
[         0 tests successful      ]
[         0 tests failed          ]

Probably worth mentioning that the testing framework is JUnit5, the tests are triggered using a template and an extension.

@TestTemplate
@ExtendWith(SomeClass.class)

There are no other functions that are annotated with @Test.

The thing is, my projects are generated from a template. All of my service are structured and behave the same. This is the only service that I ran into this issue.

I thought this has something to do with the classpath, but I see in my app.jar that it is ok, manifest looks fine, it has all the dependencies, main class is correct, and test classes are present. But still, I have tried changing java jar to java cp and state the directories of the classpath, and I passed the main class as well -

java -cp "app.jar:lib/*:classes" tests.TestsRunner -s verification -o reports

But still the same result.

Any suggestion will be great, thanks.