pythonpytestassertion

Run all PyTest assertions even with multiple failures (No option to split tests)


I'm working with PyTest to test system readiness for application installation and I want to ensure all assertions in a test run, even if some of them fail. The main reason for that these tests are run by support team which are not technical so they wanted to know about all missing items in system before they proceed for installations. Unfortunately, I don't have the option to break down the test into separate ones for each assertion. I tried multiple options like --maxfail=n but this option is not tests included in test suites not for single test. Test rerun is not a option also, as it always give same kind of failure.

Is there a recommended way to achieve this behavior in PyTest?

I am using following version:

import subprocess

import pytest

def test_executable_script():
    result = subprocess.check_output([executable_script], encoding="UTF-8")
    assert result != "", f"Error: {result}"

    result_2 = subprocess.check_output([another_executable_script], encoding="UTF-8")
    assert result != "", f"Error: {result}"


Solution

  • You can create a list for failures and assert on that list. Here is how to achieve it.

    import subprocess
    import pytest
    
    
    def test_executable_script():
        _fails = []
    
        result = subprocess.check_output([executable_script], encoding="UTF-8")
     
        if result == "":
            _fails.append("Error1")
    
        result_2 = subprocess.check_output([another_executable_script],encoding="UTF-8")
    
        if result_2 == "":
            _fails.append("Error2")
    
        assert not _fails, "Test failed because of: {}".format(" - ".join(_fails))