google-cloud-platformpathgoogle-cloud-composergoogle-cloud-logging

Set up Structured logging in cloud composer


I am trying to implement a custom logging format which includes a few variables which I get from the Google Cloud Composer environment, I have followed the docs which helped me in formatting the required environment variables by overriding the airflow.cfg logging_config_class variable. However, the composer restricts the modification of that logging class.Is there any other way to have a custom logger?

I want to be able to send a jsonPayload into google cloud which I can query. I have even tried using python gcloud loggers as follows:

from google.cloud import logging as gcloud_logging
LOG_NAME = 'airflow-worker'
client = gcloud_logging.Client()

gcloud_logger = client.logger(LOG_NAME)


gcloud_logger.log_struct({
            "message": "message text to appear in StackDriver",
            "custom_key": "value"
        }, severity='ERROR')

I followed this answer too but did not help. All these logs just show up as textPayload on gcs.


Solution

  • Google cloud composer has no support for this as of today