apache-sparkairflowspark-submit

KubernetesPodOperator Not Sending Arguments as expected


I have airflow running KubernetesPodOperator in order to do a Spark-submit call:

spark_image = f'{getenv("REGISTRY")}/myApp:{getenv("TAG")}'
j2g = KubernetesPodOperator(
    dag=dag,
    task_id='myApp',
    name='myApp',
    namespace='data',
    image=spark_image,
    cmds=['/opt/spark/bin/spark-submit'],
    configmaps=["data"],
    arguments=[
        '--master k8s://https://10.96.0.1:443',
        '--deploy-mode cluster',
        '--name myApp',
        f'--conf spark.kubernetes.container.image={spark_image}',
        'local:///app/run.py'
    ],

However, I'm getting the following error:

Error: Unrecognized option: --master k8s://https://10.96.0.1:443

Which is weird, because when I bin/bash to a running pod and execute the spark-submit command, it works.

Any idea how to pass the arguments as expected?


Solution

  • Solution from GitHub ticket: Parameter should be sent like:

     '--master=k8s://https://10.96.0.1:443',