I have an Airlfow instance spinning in Azure Kubernetes Service. Deployed with official Helm chart. And I noticed CeleryExecutor is used there. I have KubernetesPodOperator tasks, and wondering how exactly CeleryExecutor executes KubernetesPodOperator? Which magic is happen out there? Why it's not breaking?
I see a task-Pod is being created in the cluster, but how is it possible?
I assume it works the same way as executing any other operations. As you can see in the description here: https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html#how-does-this-operator-work,
Airflow communicates with the Kubernetes API (control plane; see - https://kubernetes.io/docs/concepts/overview/kubernetes-api/) using the Kube Python Client, which allows it to spin up additional Kubernetes resources to run various types of work.
If you have Airflow deployed in Kubernetes itself and use the same cluster to run the job, it simplifies communication and integration between the two systems a bit