pythonpython-3.xlogging

Extract the extra fields in logging call in log formatter


so I can add additional fields to my log message like so

logging.info("My log Message", extra={"someContext":1, "someOtherContext":2})

which is nice

but unclear how to extract all the extra fields in my log formatter

def format(self, record):
  record_dict = record.__dict__.copy()
  print(record_dict)

in the above i can see all my extra fields in the output dict but they are flattened in to a dict with load of other junk i dont want

{'name': 'root', 'msg': 'My log Message', 'args': (), 'levelname': 'INFO', 'levelno': 20, 'pathname': '.\\handler.py', 'filename': 'handler.py', 'module': 'handler', 'exc_info': None, 'exc_text': None, 'stack_info': None, 'lineno': 27, 'funcName': 'getPlan', 'created': 1575461352.0664868, 'msecs': 66.48683547973633, 'relativeCreated': 1253.0038356781006, 'thread': 15096, 'threadName': 'MainThread', 'processName': 'MainProcess', 'process': 23740, 'someContext': 1, 'someOtherContext':2}

is there any way of getting all my extra keys without having to know them all upfront,

i am writing a json formatter and want to create a dict a la

justMyExtra = ?????
to_log = {
"message" record_dict["message"], 
**justMyExtra
}

Solution

  • If you read the source code of the logging.Logger.makeRecord method, which returns a LogRecord object with the given logging information, you'll find that it merges the extra dict with the __dict__ attribute of the returing LogRecord object, so you cannot retrieve the original extra dict afterwards in the formatter.

    Instead, you can patch the logging.Logger.makeRecord method with a wrapper function that stores the given extra dict as the _extra attribute of the returning LogRecord object:

    def make_record_with_extra(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None):
        record = original_makeRecord(self, name, level, fn, lno, msg, args, exc_info, func, extra, sinfo)
        record._extra = extra
        return record
    
    original_makeRecord = logging.Logger.makeRecord
    logging.Logger.makeRecord = make_record_with_extra
    

    so that:

    class myFormatter(logging.Formatter):
        def format(self, record):
            print('Got extra:', record._extra) # or do whatever you want with _extra
            return super().format(record)
    
    logger = logging.getLogger(__name__)
    handler = logging.StreamHandler()
    handler.setFormatter(myFormatter('%(name)s - %(levelname)s - %(message)s - %(foo)s'))
    logger.addHandler(handler)
    logger.warning('test', extra={'foo': 'bar'})
    

    outputs:

    Got extra: {'foo': 'bar'}
    __main__ - WARNING - test - bar
    

    Demo: https://repl.it/@blhsing/WorthyTotalLivedistro