Understanding interferences between python importing policy and pytest mocking

I am having some troubles in understanding how to properly mock a function using pytest-mock module.

I will report a minimal reproducible example:

file: src/mini_handler.py

import tempfile

from src.mini_pdf_handler import mini_pdf_handler


def mini_handler():
    tmp_file = tempfile.NamedTemporaryFile()
    mini_pdf_handler()
    return tmp_file.name

file: src/mini_pdf_handler.py

import tempfile


def mini_pdf_handler():
    tmp_file = tempfile.NamedTemporaryFile()
    return tmp_file.name

file: tests/test_handler.py

def test_mini_handler(mocker):
    mock_tempfile = mocker.MagicMock()
    mock_tempfile.return_value.name = 'outputs/output.pdf'

    mocker.patch("src.mini_handler.tempfile.NamedTemporaryFile", side_effect=mock_tempfile)

    mini_handler()

The problem is that the mock works, but it is mocking even the NamedTemporaryFile in the mini_pdf_handler module, when it should mock it only in the mini_handler. I have the feeling the problem might be in the import policy that python have, since once a module is imported, it won't import it again. At the same time, I feel that the problem might be some silly oversight. Can someone help me?

Solution

The problem here is that you're patching "too deep".

Each module in src/ has a reference to the tempfile module. You could mock src.mini_handler.tempfile and it would not impact mini_pdf_handler, but you're dereferencing src.mini_handler.tempfile by mocking ...tempfile.NamedTemporaryFile, and in both modules this refers to the same thing.

You can fix it like this:

from src.mini_handler import mini_handler

def test_mini_handler(mocker):
    mock_tempfile = mocker.MagicMock()
    mock_tempfile.return_value.name = 'outputs/output.pdf'

    res = mocker.patch('src.mini_handler.tempfile')
    res.NamedTemporaryFile = mock_tempfile
    mini_handler()

Here we are replacing src.mini_handler.tempfile with a mock object. This doesn't impact the reference to tempfile in src.mini_pdf_handler. We then configure the NamedTemporaryFile attribute of our mock object.

In response to your comment:

Imagine we have the following dictionaries:

>>> tempfile = {'foo':'bar'}
>>> mini_handler = {'tempfile': tempfile}
>>> mini_pdf_handler = {'tempfile': tempfile}

If I modify `mini_handler['tempfile']['foo']...

>>> mini_handler['tempfile']['foo'] = 'qux'

...then that change is visible in both mini_handler['tempfile'] and mini_pdf_handler['tempfile'], because both of these names refer to the same thing:

>>> mini_handler['tempfile']['foo']
'qux'
>>> mini_pdf_handler['tempfile']['foo']
'qux'

On the other hand, if I replace mini_handler['tempfile'] with a different dictionary:

>>> mini_handler['tempfile'] = {'something': 'else'}

This is no longer the case:

>>> mini_handler['tempfile']
{'something': 'else'}
>>> mini_pdf_handler['tempfile']
{'foo': 'qux'}

The situation with modules and mocking in your test is exactly analagous to this example. Does that help?