airflowjinja2kubernetespodoperator

dynamically generate cmds in KubernetesPodOperator with Jinja2 template

I have a conf json looks like this

{ "customer": "customer1",
  "region": "south"
}

I want generate a cmds: ["aws", "domain_name", "etl_name", "--customer", "customer1", "--region", "south"]

The Jinja2 template I used:

task1 = KubernetesPodOperator(
image="....",
env_vars={...},
cmds=["aws", "domain_name", "etl_name",
"""'--customer', '{{ dag_run.conf.customer }}', '--region', '{{ dag_run.conf.region }}'""" 
)

I get the exact cmds '--customer', 'customer1', '--region', 'south'.

But when I run the code in airflow. I saw '--customer,' 'customer1,' '--region,' 'south'

How can I achieve that?

Solution

  • The problem is that you are sending the jinja parameters as one string.

    Separate each one in the command args.

        task1 = KubernetesPodOperator(
    image="....",
    env_vars={...},
    cmds=["aws", "domain_name","etl_name",
"--customer", "{{ dag_run.conf.customer }}", "--region","{{ dag_run.conf.region }}"])