I have this luigi task with its required input being 'test_file.txt'. I want to change the required input through testing the class. I need to change the input so that I can test the functionalities of the class with a file. I tried the code below, but the printed result is still the initial path-'test_file.txt'. How can I change the path only in testing? (to get 'data.json' as the printed result.
import pytest
import luigi
class LuigiToBeTested(luigi.ExternalTask):
def requires(self):
return luigi.LocalTarget("test_file.txt")
def test_Luigi():
class_instance = LuigiToBeTested()
class_instance.requires().path = 'data.json'
print('/////', class_instance.requires().path) #to get data.json
Here are 3 solutions for you
test_luigi_1
- Intercept (via function) all calls to initialize a new object of luigi.LocalTarget
and replace the input file name.
test_luigi_2
- Same with 1, but via subclass
test_luigi_3
- Instead of patching as done in 1 and 2, redesign your source code to be testable by allowing dependency injection. This is the more extensible way and also more future proof.
a. Ask for things, Don't look for things (aka Dependency Injection / Law of Demeter)
b. Fixing the Client API: Dependency Injection
c. And many many more references on why this is the preferred way.
import luigi
import pytest
class LuigiToBeTested(luigi.ExternalTask):
def requires(self):
return luigi.LocalTarget("test_file.txt")
class LuigiToBeTested2(luigi.ExternalTask):
def requires(self, file_name):
return luigi.LocalTarget(file_name)
@pytest.fixture
def amend_local_target_func_based(mocker): # The <mocker> fixture is from pytest-mock. Install via <pip install pytest-mock>.
orig = luigi.LocalTarget
def fake(file_name):
print(f"Intercepted initialization of luigi.LocalTarget via function. The file {file_name} might be replaced.")
# If you only want to replace 'test_file.txt' but not the other files, then use this.
# Otherwise, remove this if and just read from 'data.json' always.
if file_name == 'test_file.txt':
file_name = 'data.json'
return orig(file_name)
mocker.patch('luigi.LocalTarget', new=fake)
@pytest.fixture
def amend_local_target_class_based(mocker):
class LocalTargetStub(luigi.LocalTarget):
def __init__(self, file_name):
print(f"Intercepted initialization of luigi.LocalTarget via subclass. The file {file_name} might be replaced.")
# If you only want to replace 'test_file.txt' but not the other files, then use this.
# Otherwise, remove this if and just read from 'data.json' always.
if file_name == 'test_file.txt':
file_name = 'data.json'
super().__init__(file_name)
mocker.patch('luigi.LocalTarget', new=LocalTargetStub)
def test_luigi_1(amend_local_target_func_based):
class_instance = LuigiToBeTested()
print('/////', class_instance.requires().path) #to get data.json
def test_luigi_2(amend_local_target_class_based):
class_instance = LuigiToBeTested()
print('/////', class_instance.requires().path) #to get data.json
def test_luigi_3():
class_instance = LuigiToBeTested2()
print('/////', class_instance.requires('data.json').path) #to get data.json
$ pytest -q -rP
================================================================================================= PASSES ==================================================================================================
______________________________________________________________________________________________ test_luigi_1 _______________________________________________________________________________________________
------------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------------
Intercepted initialization of luigi.LocalTarget via function. The file test_file.txt might be replaced.
///// data.json
______________________________________________________________________________________________ test_luigi_2 _______________________________________________________________________________________________
------------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------------
Intercepted initialization of luigi.LocalTarget via subclass. The file test_file.txt might be replaced.
///// data.json
______________________________________________________________________________________________ test_luigi_3 _______________________________________________________________________________________________
------------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------------
///// data.json
3 passed, 1 warning in 0.07s