pythonmongodbtestingautomated-testspytest

Pytest- use a generator for mark.parametrize


I have a mongo db with a very large collection that I need to run tests on with Pytest. I am trying to do it the usual route of using the mark.parametrize dectorator but with pymongo.cursor Cursor object:

def get_all_data():
    return db["collection"].find({}) # query to retrieve all documents from the collection

@pytest.mark.parametrize("doc", get_all_data())
def test_1(doc):
    assert doc["val"] == 1
    ....

The problem with this code is pytest in the collection stage before running tests automatically converts the generator into a list. I don't want this because of 2 reasons:

  1. This is very slow due to the fact the collection is very large.
  2. Stack overflow- Not enough RAM to load all of this data anyway.

Meaning I cannot use mark.parametrize, however how can I still use a generator to run tests 1 at a time and not to load everything immediately into memory? Is it even possible with Pytest?


Solution

  • I can think of this workaround - write a fixture to pass the generator to a single test. Then check each entry individually in the same test using pytest-check (because i guess you need to assert each entry separately and continue even if some entries fail).

    @pytest.fixture
    def get_all_data():
        yield db["collection"].find({})
    
    def test_1(get_all_data):
        for each in get_all_data:
            check.is_(each["val"], 1)