pythongoogle-cloud-platformgoogle-cloud-composergoogle-cloud-error-reporting

Google Cloud Composer call an API on failure

I have a DAG running on the Google Cloud Composer. This DAG is executing some operations on top of our BigQuery (we are using airflow.contrib.operators.bigquery_operator.BigQueryOperator) I'd like to call the Error Reporting API if there is a failure in the DAG, however, I don't know how to do it.

Solution

  • The answer is using the on_failure_callback parameter in the DAG args. This is a very good starter-level article on how to use it: https://forum.astronomer.io/t/how-can-i-set-my-on-failure-and-on-success-callbacks-to-have-the-task-context/182