djangoamazon-web-servicesamazon-cloudwatchaws-cloudwatch-log-insightsdjango-logging

Log messages from Django application not uploaded in AWS CloudWatch


I have added log messages in my Django application and it was successfully logging log messages to the log file.

Now, I tried to add log messages to AWS CloudWatch. When I run the application it creates log group in AWS CloudWatch but log stream is not created and log messages also not uploaded.

I have also manually created log stream in AWS CloudWatch but still log messages were not uploaded.

I have the following logging configuration in my Django application.

logger_boto3_client = boto3.client(
    "logs",
    aws_access_key_id=CLOUDWATCH_AWS_ID,
    aws_secret_access_key=CLOUDWATCH_AWS_KEY,
    region_name=AWS_DEFAULT_REGION,
)

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'simple': {
            'format': "[cid: %(cid)s] [%(asctime)s.%(msecs)03d] %(levelname)s [%(name)s:%(lineno)s] [%(funcName)s] %(message)s",
            'datefmt': '%Y-%m-%d %H:%M:%S',
        },
    },
    'handlers': {
        'logger': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'filename': str(BASE_DIR) + '/logs/test.log',
            'formatter': 'simple',
            'filters': ['correlation'],
        },
        'watchtower': {
            "level": "DEBUG",
            "class": "watchtower.CloudWatchLogHandler",
            "boto3_client": logger_boto3_client,
            "log_group": "DemoLogs2",
            # Different stream for each environment
            "stream_name": "logs",
            "formatter": "simple",
            'filters': ['correlation'],
        }
    },
    'filters': {
        'correlation': {
            '()': 'cid.log.CidContextFilter'
        },
    },
    'loggers': {
        'root': {
            'handlers': ['logger', 'watchtower'],
            'level': 'DEBUG',
            'filters': ['correlation'],
            'propagate': True,
        }
    }
}

In my application I'm using logger like this,

import logging


logger = logging.getLogger(__name__)
logger.info("log message.")

My aws cloudwatch console.

enter image description here


Solution

  • The issue is with calling the logger in the API.

    LOGGING = {
        'version': 1,
        'disable_existing_loggers': False,
        'formatters': {
            'default': {
                'format': "[cid: %(cid)s] [%(asctime)s.%(msecs)03d] %(levelname)s [%(name)s:%(lineno)s] [%(funcName)s] %(message)s",
                'datefmt': '%Y-%m-%d %H:%M:%S',
            }
        },
        'handlers': {
            'file': {
                'level': 'DEBUG',
                'class': 'logging.handlers.RotatingFileHandler',
                'filename': str(BASE_DIR) + '/logs/test.log',
                'formatter': 'default',
                'filters': ['correlation'],
            }
            'cloudwatch': {
                "level": "DEBUG",
                "class": "watchtower.CloudWatchLogHandler",
                "boto3_client": logger_boto3_client,
                "log_group": CLOUDWATCH_LOG_GROUP,
                "stream_name": f"{ENV}-{CLOUDWATCH_DEFAULT_LOG_STREAM_NAME}",
                "formatter": "default",
                'filters': ['correlation'],
            }
        },
        'filters': {
            'correlation': {
                '()': 'cid.log.CidContextFilter'
            },
        },
        'loggers': {
            # previously named as 'root'
            'default': {
                'level': 'DEBUG',
                'handlers': ['file', 'cloudwatch'],
                'filters': ['correlation'],
                'propagate': False,
            },
        }
    }
    

    In the API,

    import logging
    
    
    # logger = logging.getLogger(__name__)
    logger = logging.getLogger("default")
    logger.info("log message.")
    

    Previously I was passing __name__ to the getLogger method. Now, I passed the logger name default to the getLogger method and now I could able to see the log messages in both the log file and cloudwatch.