pythonloggingdatabricks

Best practice to set up loggers with two different outputs in Databricks


I want to set up a logger in Databricks that does two things everytime its called:

  1. Prints the item like a regular print() command (eg "This is a log")
  2. Writes the log to a log file notebook_name_{date}.log with more granular info (eg "2024-09-12 12:51:49,694 - main - INFO - This is a log")

The goal would be to just call this once, similar to logger.info('log'), so if it needs to be wrapped in a function that is fine.

This is my code so far but it prints the timestamp, name, etc in the cell output which I don't want:

import logging
from datetime import datetime


def get_current_date():
    # Get the current date and time
    now = datetime.now()

    # Format the date as DD Uppercase Three Letter Month YEAR
    formatted_date = now.strftime("%d%b%Y").upper()
    return formatted_date


log_file = f'logfile_{formatted_date}.log'


# Create a logger
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

# Create a file handler
file_handler = logging.FileHandler(log_file)

# Create a stream handler
stream_handler = logging.StreamHandler()

# Create a formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)
stream_handler.setFormatter(formatter)

# Add the handlers to the logger
logger.addHandler(file_handler)
logger.addHandler(stream_handler)

# Test the logger
logger.info('This is a test!!!')
logger.info('this is a test again')



Solution

  • //etc in the cell output which I don't want://

    Create another formatter variable for Streamingoutput.

    sformatter = logging.Formatter('%(message)s')
    stream_handler.setFormatter(sformatter)
    

    This way it will display only the message in Databricks and Log file will have more details.