I am trying to build a strategy to log from Luigi in such a way that there is a configurable list of outputs including stdout and a custom list of files. I would like to be able to set the logging level at runtime. Our system uses Luigi to call spark from Jenkins. Thank you in advance.
Inside any of the Task class methods, you can do:
class Agg(luigi.Task):
_date = luigi.DateParameter()
def output(self):
return luigi.LocalTarget("file_%.txt" % self._date)
def run(self):
# Use the luigi-interface to log to console
logger = logging.getLogger('luigi-interface')
logger.info("Running --> Agg.Task")