pythonpython-hypothesis

Using pytest and hypothesis, how can I make a test immediately return after discovering the first counterexample?

I'm developing a library, and I'm using hypothesis to test it. I usually sketch out a (buggy) implementation of a function, implement tests, then iterate by fixing errors and running tests. Usually these errors are very simple (e.g., a typo), and I don't need simplified test cases to figure out the issue. For example:

def foo(value):
    return vslue + 1 # a silly typo

@given(st.integers())
def test_foo(x):
    assert foo(x) == x + 1

How can I make hypothesis stop generating test cases as soon as it has found a single counterexample? Ideally, this would be using commandline flags to pytest.

Solution

  • Building on my comment from earlier - did you look at this?

    https://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.phases

    Looks like there's some support for having different "Setting Profiles". I bet you could have one where you eliminate the Shrinking phase. Maybe that's the way to do.

    Yeah! That's probably it :) That gets you this command line option:

    pytest tests --hypothesis-profile <profile-name>

    So, in summary:

    In your conftest.py:

    from hypothesis import settings, Phase
settings.register_profile("failfast", phases=[Phase.explicit, Phase.reuse, Phase.generate])

    And then run it with: pytest tests --hypothesis-profile failfast