dockerkubernetesairflowdockeroperatorkubernetesexecutor

Run docker containers in Airflow hosted in kubernetes


I have been trying to deploy airflow using the helm chart (as a subchart) on a kubernetes cluster that is hosted in my homelab.

I am facing some issues when migrating the dags I have been testing locally (on docker-compose) to the kubernetes cluster. when you use airflow on kubernetes, is it possible to use the CeleryExecutor? if so, how should I change the DockerOperators in my dags? When trying to find an answer, i dtumbled upon the following stackoverflow post suggesting that using the DockerOperator is not a good choice when running on kubernetes , is that correct?


Solution

    1. Can you use CeleryExecutor on Kubernetes?

    Yes, you can. It’s supported and works fine in k8s. You’ll just need:

    But most people go with the KubernetesExecutor with KubernetesPodOperator instead, because they:

    1. Should you avoid DockerOperator in Kubernetes?

    Yes, generally.

    Why?

    Alternatives: Use KubernetesPodOperator instead. It:

    1. Does this mean you shouldn’t use CeleryExecutor?

    Not necessarily. You can continue using it, but you might consider migrating to:

    It depends on your workloads. If you’re already comfortable with Celery, and need specific control over parallelism, queues, or workers—keep it.

    Read more about KubernetesPodOperator.