I'm doing functional testing and using Luigi to run a monthly rollup task. There is separate container for performing tasks and one container for database. This task only runs its dependent tasks, which are 30 daily rollup tasks. These daily rollup tasks run 24 hourly tasks.
The daily rollup task fails sometimes, both in the pipeline and locally. Whenever it fails, it throws this error:
taskrunner: Traceback (most recent call last):
taskrunner: File "/usr/lib/python3.11/site-packages/luigi/worker.py", line 188, in run
taskrunner: raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
taskrunner: RuntimeError: Unfulfilled dependencies at run time:.........
I can't share the complete error. :(
The error indicates that the task failed due to an unfulfilled dependency. However, all hourly rollup tasks are completed with the status as DONE. This issue does not occur every time; sometimes it passes.
The problem was with the resources. When I was running multiple apps and run the tests it fails. In the pipeline I fixed it by adding extra CPU and RAM to the container. This documentation explains how to configure run time for docker container.
Runtime options with Memory, CPUs, and GPUs