bashairflow

How to run a shell script through apache airflow

I am trying to run a shell script through airflow, the shell script works when I execute it locally.

This is my Dag code:

    dag = DAG(dag_id='Phase1_dag_v1', default_args=args, schedule_interval=None)

with dag:
    test= BashOperator(
        task_id='test',
        bash_command='bash -i /home/lnxuser/airflow/files/test_ns/env.sh'
    )
    test

This is the error log from airflow:

[2021-02-04 19:21:08,127] {taskinstance.py:1396} ERROR - bash -i /home/lnxuser/airflow/files/pam_ns/env.sh
Traceback (most recent call last):
  File "/home/lnxuser/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1086, in _run_raw_task
    self._prepare_and_execute_task_with_callbacks(context, task)
  File "/home/lnxuser/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1224, in _prepare_and_execute_task_with_callbacks
    self.render_templates(context=context)
  File "/home/lnxuser/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1684, in render_templates
    self.task.render_template_fields(context)
  File "/home/lnxuser/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 857, in render_template_fields
    self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
  File "/home/lnxuser/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 870, in _do_render_template_fields
    rendered_content = self.render_template(content, context, jinja_env, seen_oids)
  File "/home/lnxuser/.local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 905, in render_template
    return jinja_env.get_template(content).render(**context)
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/lib/python3/dist-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/usr/lib/python3/dist-packages/jinja2/loaders.py", line 113, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/usr/lib/python3/dist-packages/jinja2/loaders.py", line 187, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: bash -i /home/lnxuser/airflow/files/pam_ns/env.sh
[2021-02-04 19:21:08,169] {taskinstance.py:1433} INFO - Marking task as FAILED. dag_id=Phase1_dag_v1, task_id=PAM, execution_date=20210204T192105, start_date=20210204T192107, end_date=20210204T192108
[2021-02-04 19:21:08,382] {local_task_job.py:118} INFO - Task exited with return code 1

I am a bit of a novice so not really sure what is going wrong, I could be using airflow in completely the wrong way.

Anyone know what's going on here?

Solution

  • You need to add space after the script name because you are not using templating and the script is not in relative path to the dag file.

    bash_command='bash -i /home/lnxuser/airflow/files/test_ns/env.sh '

    The explanation when space is needed can be found in the docs