I'd like to be able to run Python's unittest module programmatically via a subprocess (e.g. subprocess.Popen(), subprocess.run(), asyncio.create_subprocess_exec()) and have it auto-discover tests.
I do not want to run the tests by importing the unittest module into my script, because I would like the same code to be able to run any arbitrary command from the command line, and I'd like to avoid handling running tests differently than other commands.
Here is a GitHub repository with code that illustrates the issue I'm seeing: https://github.com/sscovil/python-subprocess
For completeness, I'll include it here as well.
.
├── src
│ ├── __init__.py
│ └── example
│ ├── __init__.py
│ └── runner.py
└── test
├── __init__.py
└── example
├── __init__.py
└── runner_test.py
src/example/runner.py
import asyncio
import os
import shutil
import subprocess
import unittest
from subprocess import CompletedProcess, PIPE
from typing import Final, List
UNIT_TEST_CMD: Final[str] = "python -m unittest discover test '*_test.py' --locals -b -c -f"
def _parse_cmd(cmd: str) -> List[str]:
"""Helper function that splits a command string into a list of arguments with a full path to the executable."""
args: List[str] = cmd.split(" ")
args[0] = shutil.which(args[0])
return args
async def async_exec(cmd: str, *args, **kwargs) -> int:
"""Runs a command using asyncio.create_subprocess_exec() and logs the output."""
cmd_args: List[str] = _parse_cmd(cmd)
process = await asyncio.create_subprocess_exec(*cmd_args, stdout=PIPE, stderr=PIPE, *args, **kwargs)
stdout, stderr = await process.communicate()
if stdout:
print(stdout.decode().strip())
else:
print(stderr.decode().strip())
return process.returncode
def popen(cmd: str, *args, **kwargs) -> int:
"""Runs a command using subprocess.call() and logs the output."""
cmd_args: List[str] = _parse_cmd(cmd)
with subprocess.Popen(cmd_args, stdout=PIPE, stderr=PIPE, text=True, *args, **kwargs) as process:
stdout, stderr = process.communicate()
if stdout:
print(stdout.strip())
else:
print(stderr.strip())
return process.returncode
def run(cmd: str, *args, **kwargs) -> int:
"""Runs a command using subprocess.run() and logs the output."""
cmd_args: List[str] = _parse_cmd(cmd)
process: CompletedProcess = subprocess.run(cmd_args, stdout=PIPE, stderr=PIPE, check=True, *args, **kwargs)
if process.stdout:
print(process.stdout.decode().strip())
else:
print(process.stderr.decode().strip())
return process.returncode
def unittest_discover() -> unittest.TestResult:
"""Runs all tests in the given directory that match the given pattern, and returns a TestResult object."""
start_dir = os.path.join(os.getcwd(), "test")
pattern = "*_test.py"
tests = unittest.TextTestRunner(buffer=True, failfast=True, tb_locals=True, verbosity=2)
results = tests.run(unittest.defaultTestLoader.discover(start_dir=start_dir, pattern=pattern))
return results
def main():
"""Runs the example."""
print("\nRunning tests using asyncio.create_subprocess_exec...\n")
asyncio.run(async_exec(UNIT_TEST_CMD))
print("\nRunning tests using subprocess.Popen...\n")
popen(UNIT_TEST_CMD)
print("\nRunning tests using subprocess.run...\n")
run(UNIT_TEST_CMD)
print("\nRunning tests using unittest.defaultTestLoader...\n")
unittest_discover()
if __name__ == "__main__":
main()
test/example/runner_test.py
import unittest
from src.example.runner import async_exec, popen, run, unittest_discover
class AsyncTestRunner(unittest.IsolatedAsyncioTestCase):
async def test_async_call(self):
self.assertEqual(await async_exec("echo Hello"), 0)
class TestRunners(unittest.TestCase):
def test_popen(self):
self.assertEqual(popen("echo Hello"), 0)
def test_run(self):
self.assertEqual(run("echo Hello"), 0)
def test_unittest_discover(self):
results = unittest_discover()
self.assertEqual(results.testsRun, 4) # There are 4 test cases in this file
if __name__ == "__main__":
unittest.main()
When running tests from the command line, Python's unittest module auto-discovers tests in the test directory:
python -m unittest discover test '*_test.py' --locals -bcf
....
----------------------------------------------------------------------
Ran 4 tests in 0.855s
OK
...but it fails to auto-discover tests when that same command is run using Python's subprocess module:
$ python -m src.example.runner
Running tests using asyncio.create_subprocess_exec...
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
Running tests using subprocess.Popen...
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
Running tests using subprocess.run...
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
Running tests using unittest.defaultTestLoader...
test_async_call (example.runner_test.AsyncTestRunner.test_async_call) ... ok
test_popen (example.runner_test.TestRunners.test_popen) ... ok
test_run (example.runner_test.TestRunners.test_run) ... ok
test_unittest_discover (example.runner_test.TestRunners.test_unittest_discover) ... ok
----------------------------------------------------------------------
Ran 4 tests in 0.864s
OK
Note that the unittest.defaultTestLoader test runner works as expected, because it is explicitly using the unittest module to run the other tests. However, when running tests using asyncio.create_subprocess_exec, subprocess.Popen, or subprocess.run, as if using the CLI from the command line, the tests are not auto-discovered.
If you have Docker installed, you can run the tests in a container using any version of Python you like. For example:
docker run -it --rm -v $(pwd):$(pwd) -w $(pwd) --name test python:3.11-alpine python3 -m src.example.runner
docker run -it --rm -v $(pwd):$(pwd) -w $(pwd) --name test python:3.10 python3 -m src.example.runner
In every version I tried, from 3.8 to 3.11, I saw the same results.
Why does Python unittest auto-discovery not work when running in a subprocess?
This has nothing to do with running in a subprocess. Your cmd_args is broken.
You wrote a command line like what you'd write in a shell, but it doesn't go through any of the processing a shell would apply. It goes through your own custom processing, where you split it on single spaces and then try to locate the executable with shutil.which.
One of the processing steps the shell would apply is quote removal, which is what would remove the ' characters from your '*_test.py' pattern if you ran that command in a shell. Because this isn't going through a shell, those characters remain in the argument, so you end up telling unittest test discovery to look for test files with ' characters at the start and end of their names.
You don't have any test files with ' characters at the start and end of their names, and such names would be incompatible with test discovery even if you had any, so test discovery finds nothing.
You need to do something that results in a valid argv list, without quotation marks in the pattern. I recommend just writing out the list manually:
cmd = [
'python',
'-m',
'unittest',
'discover',
'test',
'*_test.py',
'--locals',
'-b',
'-c',
'-f',
]
Alternatively, you could keep your current command line processing and just remove the ' characters from your UNIT_TEST_CMD, but with how shell-like your command looks, it's too easy to get mixed up about the syntax you're using.
It's also possible to just invoke a shell to process your command line with asyncio.create_subprocess_shell, or using shell=True with subprocess, but it's way too easy to create subtle security holes when relying on shell processing.