How to send AWS MWAA cluster metrics to DataDog?
AWS's Managed Workflows for Apache Airflow (MWAA) provides cluster metrics like the CPUUtilization and MemoryUtilization for the BaseWorker, Scheduler and WebServer. I would like to have these in DataDog so I can set usage alerts. Here's a example of what they look like in Cloudwatch:
However, DataDog's official MWAA integration doesn't have these metrics listed. Is there a way to directly specify the metrics and forward them to DataDog? Or, possibly, are they already being picked up but under an obscure namespace?
After some research I found that you need to activate an option to send custom metrics from AWS.
You have to follow the next steps:
Datadog UI → Integrations → Search for AWS → Select your account → Go to Metric Collection → Turn on Collect custom metrics from Cloudwatch
After some minutes search for something like aws.mwaa.cpuutilization and it should be there for you to filter by cluster or any other type of query
- How to get jobId that was submitted using Dataproc Workflow Template
- External files in Airflow DAG
- Airflow Sync Dags from S3 Causes Dag Execution Error
- Airflow - How to pass xcom variable into Python function
- Connection error from GCP Composer (Airflow) to GCP PubSub
- airflow TriggerDagRunOperator how to change the execution date
- Access Forbidden while accessing log in airflow with CeleryExecutor
- How to manage python packages between airflow dags?
- Passing Parameters from ECS to Airflow XCom
- jinja2.exceptions.TemplateNotFound error with airflow bash operator
- [Airflow]: Dynamic Task Mapping on DockerOperator using Xcoms
- Airflow UI parameters not being passing on to DAG
- How to Trigger a DAG on the success of a another DAG in Airflow using Python?
- Airflow retry of multiple tasks as a group
- Airflow timetable that combines multiple cron expressions?
- Airflow: Can't connect to ('0.0.0.0', 8080)
- Airflow - Stop DAG based on condition (skip remaining tasks after branch)
- Airflow webserver pod unable to get log from worker pod
- Datasets tab not showing in Airflow
- Altering "readtimeout" for Lambda Invoke in AirFlow
- Dynamic Task Mapping for http operator and query string
- Airflow DAG to retry if any of the Airflow Tasks fails
- Calling a Google cloud function within Airflow DAG
- Why function on_failure_callback for Airflow dag not running on task fail?
- Airflow ERROR - Error authorizing OAuth access token: Invalid JSON Web Key Set. [The request to sign in was denied.]
- How to combine multiple DAGs in Airflow
- Limiting concurrency of parallel task execution
- How to unpause Airflow DAG without triggering
- How to know what caused a simple DAG in Apache Airflow to fail?
- Airflow BashOperator Parameter From XCom Value