airflowwebhooksfreshdesk

Triggering Apache Airflow's data pipeline using an external event in Freshdesk?


I have created a data pipeline in Airflow which helps to update the tickets in Freshdesk whenever some values are change in the tickets during business hours. My logic for updatding the ticket is working fine, but I want to make the trigger the dags whenever a event is occured in Freshdesk interface so that I don't need to manually trigger the dags.

There are automation rules in FreshDesk which provides the external webhook in https://companyname.freshdesk.com/api/v2/tickets/{{ticket.id}} where {ticket.id} is essentail to know the id for which the tickets are updated from the data pipeline.

I need to know how to connect the external trigger such as webhook so that we can trigger the data-pipeline automatically in Airflow.

My Airflow Version 1.10


Solution

  • Airflow doesn't have a mechanism that allows triggering DAG based on webhooks from other services.

    That use case might be covered by AIP-35 Add Signal Based Scheduling To Airflow but this is currently a draft idea of enhancement to Airflow.

    To achieve your use case with current functionality you will need to other application/service to handle the webhook fired by Freshdesk and triggering a new DAG run in Airflow with RestAPI Trigger new DAG run endpoint

    You need to be aware of frequency. If the frequency of creating new DAG runs is too high you should consider switching to a batch solution (which Airflow is more suitable for)