python-2.7jenkinsluigi

How to log from Python Luigi


I am trying to build a strategy to log from Luigi in such a way that there is a configurable list of outputs including stdout and a custom list of files. I would like to be able to set the logging level at runtime. Our system uses Luigi to call spark from Jenkins. Thank you in advance.


Solution

  • Inside any of the Task class methods, you can do:

    class Agg(luigi.Task):
      _date = luigi.DateParameter()
    
      def output(self):
        return luigi.LocalTarget("file_%.txt" % self._date)
    
      def run(self):
        # Use the luigi-interface to log to console
        logger = logging.getLogger('luigi-interface')
        logger.info("Running --> Agg.Task")