Parameterizing an entire Pytest session

I'm looking to test a Python module using pytest which uses data from an external git submodule. I want to include an (eventually opt-in) option to test my code against many different versions of this submodule automatically, or in other words running all of my tests against each git tag >= 1.0.0. Essentially I would like calling pytest once to expand to the following commands:

git checkout 1.0.0
pytest
git checkout 1.0.1
pytest
git checkout 1.0.2
pytest
...

Where the number of tests run in that session is amount of tests * number of submodule versions. After taking a look at the documentation for pytest hook functions, I found pytest_sessionstart(), and I am able to dynamically pick a git version tag before running the suite:

@pytest.hookimpl()
def pytest_sessionstart(session):
    # Grab and populate the repo
    repo = git.Repo(path_to_repo)
    repo.git.fetch()

    tag_list = sorted([version_string_to_tuple(tag.name) for tag in repo.tags])
    # Only select versions >= 1.0.0
    tag_list = tag_list[tag_list.index((1, 0, 0)):]
    
    # Checkout (singular) version
    repo.git.checkout(version_tuple_to_string(tag_list[-1])) # latest, because why not

    # Test collection/execution begins after this function ends

But this will only allow me to checkout one version per pytest invocation.

Solution

I have thought about your problem - a pretty interesting situation. I have never run into this before, so I’m not fully confident my solution is the most correct, but it seems to me that your original idea doesn’t quite fit the way pytest works.

What I would suggest instead is to handle this outside of pytest by writing a simple script that loops over your git tags, checks out each one, and runs pytest seprately.

Why I think the original idea doesn’t quite fit:

The reason I would not try to handle this directly inside pytest_sessionstart is that pytest is fundamentally designed to set up one consistent environment for the entire test session. It collects all tests once, builds fixtures, and then runs them. If you change your underlying code in the middle of the session, it could cause inconsistencies or undefined behavior.

What I suggest doing instead:

I would lean toward keeping it simple and just driving multple pytest runs externally.

You can write a small script that loops over the git tags you care about, does a git checkout, and then invokes pytest. That way, each pytest session is completely independent.

For example, in bash:

for tag in $(git tag --sort=v:refname | grep -E '^1\.[0-9]+(\.[0-9]+)?$'); do
    echo "Checking out this tag: $tag"
    git checkout $tag
    pytest
done

Or in Python:

import subprocess

tags = subprocess.check_output(['git', 'tag', '--sort=v:refname'], text=True).splitlines()
# You can add any filter, for example >= 1.0.0
tags = [tag for tag in tags if tag >= '1.0.0']

for tag in tags:
    print(f"\n==> Testing tag: {tag}")
    subprocess.run(['git', 'checkout', tag], check=True)
    subprocess.run(['pytest'], check=True)

This approach has the advantage of not touching your actual pytest config at all. Your pytest_sessionstart stays simple, and your tests assume the current working directory has the right version of the code.