I'm more familiar with the google test framework and know about the primary behavior pair they support about ASSERT_*
vs EXPECT_*
which are the fatal and non-fatal assert modes.
From the documentation:
The assertions come in pairs that test the same thing but have different effects on the current function.
ASSERT_*
versions generate fatal failures when they fail, and abort the current function.EXPECT_*
versions generate nonfatal failures, which don't abort the current function. Usually EXPECT_* are preferred, as they allow more than one failures to be reported in a test. However, you should use ASSERT_* if it doesn't make sense to continue when the assertion in question fails.
Question: does pytest also have a non fatal assert flavor or mode I can enable?
It's nice to allow a full range of tests to maximally execute to get the richest failure history rather than abort at the first failure and potentially hide subsequent failures that have to be discovered piecewise by running multiple instances of the test application.
There is no feature like that in pytest
. Plain assert
statements will fail the test immediately when the result of an expression is falsey. For a non-fatal assertion, you could use a function call - the answer from hoefling mentions a 3rd-party pytest plugin providing such a function. You could easily recreate the EXPECT_* idea without using a plugin by appending to a list of errors and then asserting the list is empty at the end of the test.
It's nice to allow a full range of tests to maximally execute to get the richest failure history rather than abort at the first failure and potentially hide subsequent failures that have to be discovered piecewise by running multiple instances of the test application.
Opinions differ on whether this is nice or not. In Python OSS, the popular approach is: write more tests, write smaller tests, and write tests that (ideally) only assert on one thing. Every potential "subsequent failure that is discovered piecewise" would be written in its own separate test. Parametrized tests can often be used to reduce repetition, where one test definition can be expanded into multiple test cases with different input data.