I'm working with Apache Airflow and need to configure a task that operates in a passive manner. Specifically, this task should use the Airflow API to update its own status, while the execution of the task itself is handled externally.
For example, the task could represent a human sign-off or an asynchronous operation where the external system performs the work. How can I set up such a task in Airflow so that it remains passive and only updates its status through the API?
That's unfortunately really unfitting use case for airflow. Basically you have 2 options:
The clear downside is - both options will occupy worker node, even though job is being offloaded.
Alternatively - you can separate jobs triggering from jobs validating status e.g. running dag-s triggering, then checking status around expected completion time from separate dag. Which although least explicit in design will be likely most efficient in computation.