python-3.xscrapylog-level

Scrapy - How to set different log levels for Stdout and File

I would like to know how to set different log levels for log on terminal/stdout vs logging into file.

I would like to have INFO level for terminal and ERROR level for logging into file.

I am updating settings.py as below:

LOG_LEVEL = 'ERROR'
LOG_FORMAT = '%(levelname)s - %(message)s'
LOG_FILE = './log.txt'

However this will set ERROR level for both terminal as well as file.

Is there a way to set different log levels for scrapy.

Solution

  • You can add multiple handlers to your logger :

    import logging

logger = logging.getLogger()

TERMINAL_LOG_LEVEL = logging.INFO
TERMINAL_LOG_FORMAT = 'your terminal log format'

# Terminal handler
terminalHandler = logging.StreamHandler()
terminalHandler.setLevel(TERMINAL_LOG_LEVEL)
terminalHandler.setFormatter(logging.Formatter(TERMINAL_LOG_FORMAT))
logger.addHandler(terminalHandler)

LOG_LEVEL = logging.ERROR # It is better to use logging's enum values
LOG_FORMAT = '%(levelname)s - %(message)s'
LOG_FILE = './log.txt'

# File handler
fileHandler = logging.FileHandler(LOG_FILE)
fileHandler.setLevel(LOG_LEVEL)
fileHandler.setFormatter(logging.Formatter(LOG_FORMAT))
logger.addHandler(fileHandler)

    For more details, check out scrapy's logging docs.

    If you still encounter a problem, post a comment !

    Wish you good luck with your project ⭐️