I'm trying to write a test suite for verifying the state of some servers using testinfra.
It's my first time working with python/testinfra/pytest.
As a brief pseudocodey example
test_files.py
testinfra_hosts=[server1,server2,server3]
with open("tests/microservices_default_params.yml", "r") as f:
try:
default_params = yaml.safe_load(f)
except yaml.YAMLError as exc:
print(exc)
with open("tests/" + server + "/params/" + server + "_params.yml", "r") as f:
try:
instance_params = yaml.safe_load(f)
except yaml.YAMLError as exc:
print(exc)
@pytest.mark.parametrize(
"name", [] + default_params["files"] + instance_params["files"]
)
def test_files(host, name):
file = host.file(name)
assert file.exists
Each server has it's own unique params yaml file. I want every server to go through the same test, however I need each server to run the test with its own parametrized values from its respective .yml file.
The problem with the code above, is that it will try to execute all of server1s unique params against both server 2 and 3, then will start again with server 2 being run against servers 1-3 unique params.
I can't find a clean way to essentially have the test run once with server1 as the host, and server 1 params, then do the same again with server2 and server2 params etc.
I've tried using for loops within the test file itself, reading each instance_params.yml into a dictionary with the key being the server name and value containing all of that servers params - but that doesn't smell very good and because the assert is inside the loop, if one of the params for that server fails the loop exits and doesn't attempt any further params for that server.
I've looked into pytest_collection_modifyitems but I can't quite get my head around how to let it do what I want. I feel like there may be an easy solution to this that I'm missing.
My last resort would be to seperate out the tests and parametrized params individually as
@pytest.mark.parametrize(
"server1_params", instance_params['server1']['files]
)
def_test_files_server1(host, server1_params):
...
@pytest.mark.parametrize(
"server2_params", instance_params['server2']['files]
)
def_test_files_server2(host,server2_params):
...
That approach doesn't sound right to me though.
Any help for a fresh junior would be appreciated, I've never asked anything here before Hope it makes sense :)
Update: @ajk Found the solution! The pytest_generate_tests function was exactly what I was needing - and I've furthered my understanding of pytest along the way.
Thanks ajk! I owe you one :D
That's a fine question, and it looks like you've come at it from a couple different angles already. I'm no expert myself, but there are a few different ways I can think to do something like this in pytest. They all involve handing the heavy lifting over to fixtures. So here's a quick breakdown of one way to adapt the code you've already shared to use some fixtures:
Default Parameters
It looks like you've got some parameters that are not host-specific. It likely makes sense to pull those into a session-scoped fixture so you can reuse the same values across many hosts:
@pytest.fixture(scope="session")
def default_params():
with open("tests/microservices_default_params.yml", "r") as f:
try:
return yaml.safe_load(f)
except yaml.YAMLError as exc:
print(exc)
That way, you can add default_params as an argument to any test function that needs those values.
Host-specific Parameters
You certainly could load these parameters as you were doing before, or put some lookup logic in the test itself. That may be the best and clearest approach! Another option is to have a parametrized fixture whose value varies by instance:
@pytest.fixture(scope="function", params=testinfra_hosts)
def instance_params(request):
with open(f"tests/{request.param}/params/{request.param}_params.yml", "r") as f:
try:
return request.param, yaml.safe_load(f)
except yaml.YAMLError as exc:
print(exc)
Now, if we add instance_params as an argument to a test function it will run the test once for each entry in testinfra_hosts. The fixture will return a new value each time, based on the active host.
Writing the Test
If we farm the heavy lifting out to fixtures, the actual test can become simpler:
def test_files(default_params, instance_params):
hostname, params = instance_params
merged_files = default_params["files"] + params["files"]
print(f"""
Host: {hostname}
Files: {merged_files}
""")
(I'm not even asserting anything here, just playing with the fixtures to make sure they're doing what I think they're doing)
Taking it for a spin
I tried this out locally with the following sample yaml files:
tests/microservices_default_params.yml
files:
- common_file1.txt
- common_file2.txt
tests/server1/params/server1_params.yml
files:
- server1_file.txt
tests/server2/params/server2_params.yml
files:
- server2_file.txt
tests/server3/params/server3_params.yml
files:
- server3_file.txt
- server3_file2.txt
Running the test file produces:
test_infra.py::test_files[server1]
Host: server1
Files: ['common_file1.txt', 'common_file2.txt', 'server1_file.txt']
PASSED
test_infra.py::test_files[server2]
Host: server2
Files: ['common_file1.txt', 'common_file2.txt', 'server2_file.txt']
PASSED
test_infra.py::test_files[server3]
Host: server3
Files: ['common_file1.txt', 'common_file2.txt', 'server3_file.txt', 'server3_file2.txt']
PASSED
Which seems to be at least the general direction you were shooting for. I hope some of that is useful - good luck and happy testing!
Update
The comment below asks about breaking this out so each file from the list generates its own test. I'm not sure the best way to do that, but a couple options might be:
@pytest.mark.parametrizepytest_generate_tests to set up dynamic parametrization, similar to what's described here.In either case, you could start with something like this:
import pytest
import yaml
testinfra_hosts = ["server1", "server2", "server3"]
def get_default_params():
with open("tests/microservices_default_params.yml", "r") as f:
try:
return yaml.safe_load(f)
except yaml.YAMLError as exc:
print(exc)
def get_instance_params(instance):
with open(f"tests/{instance}/params/{instance}_params.yml", "r") as f:
try:
return yaml.safe_load(f)
except yaml.YAMLError as exc:
print(exc)
To use @pytest.mark.parametrize, you could follow that up with:
def get_instance_files(instances):
default_files = get_default_params()["files"]
for instance in instances:
instance_files = default_files + get_instance_params(instance)["files"]
for filename in instance_files:
yield (instance, filename)
@pytest.mark.parametrize("instance_file", get_instance_files(testinfra_hosts))
def test_files(instance_file):
hostname, filename = instance_file
print(
f"""
Host: {hostname}
Files: {filename}
"""
)
Or to take the pytest_generate_tests approach, you could do this instead:
def pytest_generate_tests(metafunc):
if "instance_file" in metafunc.fixturenames:
default_files = get_default_params()["files"]
params = [
(host, filename)
for host in testinfra_hosts
for filename in (
default_files + get_instance_params(host)["files"]
)
]
metafunc.parametrize(
"instance_file", params, ids=["_".join(param) for param in params]
)
def test_files(instance_file):
hostname, filename = instance_file
print(
f"""
Host: {hostname}
Files: {filename}
"""
)
Either way could work, and I suspect more experienced folks might package the pytest_generate_tests version up into a class and clean up the logic a bit. We have to start somewhere though, eh?