Filter Different log level messages to different log files

So, I am trying to create a logging system for my Django Project where I need to save all the different log level messages to different files.

TLDR,

I managed to get the Logs for particular level to appear in their respective files, but

• Debug.log contains all the log level messages

• Info.log contains all log level messages leaving debug

• warning.log contains WARN, ERROR & CRITICAL log level messages

• error.log contains ERROR & CRITICAL log level messages

• critical.log contains just CRITICAL log level messages

So, I followed the official https://docs.djangoproject.com/en/2.2/topics/logging/

Got a clearer picture from this Django rest framework logging different levels on different files

Then wrote the following code.

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'debug_logs': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': 'logs/debug.log',
            'formatter': 'verbose',
        },
        'error_logs': {
            'level': 'ERROR',
            'class': 'logging.FileHandler',
            'filename': 'logs/error.log',
            'formatter': 'verbose',
        },
        'warn_logs': {
            'level': 'WARN',
            'class': 'logging.FileHandler',
            'filename': 'logs/warn.log',
            'formatter': 'verbose',
        },
        'info_logs': {
            'level': 'INFO',
            'class': 'logging.FileHandler',
            'filename': 'logs/info.log',
            'formatter': 'verbose',
        },
        'critical_logs': {
            'level': 'CRITICAL',
            'class': 'logging.FileHandler',
            'filename': 'logs/critical.log',
            'formatter': 'verbose',
        },

    },
    'loggers': {
        '': {
            'handlers': ['error_logs', 'warn_logs', 'info_logs', 'critical_logs', 'debug_logs'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

So, now the result that i am getting is as follows,

debug.log

INFO 2019-05-02 05:36:22,888 autoreload 1683 4558792128 Watching for file changes with StatReloader ERROR 2019-05-02 05:36:26,604 getQuestions 1683 123145336807424 error! DEBUG 2019-05-02 05:36:26,604 getQuestions 1683 123145336807424 debug INFO 2019-05-02 05:36:26,604 getQuestions 1683 123145336807424 info WARNING 2019-05-02 05:36:26,605 getQuestions 1683 123145336807424 warning CRITICAL 2019-05-02 05:36:26,605 getQuestions 1683 123145336807424 critical

info.log

INFO 2019-05-02 05:36:22,888 autoreload 1683 4558792128 Watching for file changes with StatReloader ERROR 2019-05-02 05:36:26,604 getQuestions 1683 123145336807424 error! INFO 2019-05-02 05:36:26,604 getQuestions 1683 123145336807424 info WARNING 2019-05-02 05:36:26,605 getQuestions 1683 123145336807424 warning CRITICAL 2019-05-02 05:36:26,605 getQuestions 1683 123145336807424 critical

warn.log

ERROR 2019-05-02 05:36:26,604 getQuestions 1683 123145336807424 error! WARNING 2019-05-02 05:36:26,605 getQuestions 1683 123145336807424 warning CRITICAL 2019-05-02 05:36:26,605 getQuestions 1683 123145336807424 critical

error.log

ERROR 2019-05-02 05:36:26,604 getQuestions 1683 123145336807424 error! CRITICAL 2019-05-02 05:36:26,605 getQuestions 1683 123145336807424 critical

critical.log

CRITICAL 2019-05-02 05:36:26,605 getQuestions 1683 123145336807424 critical

As you mentioned in your question, DEBUG contains all the log level messages, INFO has contains all log level messages except for DEBUG and so on. That is how log level works, DEBUG is the level highest, and CRITICAL is the lowest. So DEBUG will contain all the other log level messages, and CRITICAL will get one. More information can be found in Python Documentation

Having different files for different log levels is not a good design to be honest. Because when you write logs in your code, it will maintain a sequence of execution. With that sequential logs, you should be able to grasp how the code executed and why the error occurred finally. There are many tools to parse error logs from a single file containing different level of logs, for example, Kibana or ELK. You can use Sentry to track error logs too.

I would recommend to maintain Django app specific loggers. So all the logs from a Django app will be in one single logger file. But its up to your application architecture/necessity on how you should set up your logger.

Finally, even if you want to get different level of logs in different files, you need to have different loggers configured, like this:

'loggers': {
    'error_logger': {
        'handlers': ['error_logs'],
        'level': 'DEBUG',
        'propagate': True,
    },
   'warn_logger': {
        'handlers': ['warn_logs'],
        'level': 'WARN',
        'propagate': True,
    }
    # in same way configure info_logs, debug_logs and so on.
}

Then in your files, you need to import loggers like this:

info_logger = logging.getLogger('info_logger')
error_logger = logging.getLogger('error_logger')

#inside code
info_logger.info("some info")
error_logger.error("error")