I have been trying to deploy airflow using the helm chart (as a subchart) on a kubernetes cluster that is hosted in my homelab.
I am facing some issues when migrating the dags I have been testing locally (on docker-compose) to the kubernetes cluster. when you use airflow on kubernetes, is it possible to use the CeleryExecutor? if so, how should I change the DockerOperators in my dags? When trying to find an answer, i dtumbled upon the following stackoverflow post suggesting that using the DockerOperator is not a good choice when running on kubernetes , is that correct?
Does this mean I shouldn't use the CeleryExecutor ?
What are the best practice when migrating dags from docker-compose to kubernetes ? Thanks
Yes, you can. It’s supported and works fine in k8s. You’ll just need:
But most people go with the KubernetesExecutor with KubernetesPodOperator instead, because they:
Yes, generally.
Why?
Alternatives: Use KubernetesPodOperator instead. It:
Not necessarily. You can continue using it, but you might consider migrating to:
It depends on your workloads. If you’re already comfortable with Celery, and need specific control over parallelism, queues, or workers—keep it.
Read more about KubernetesPodOperator.