airflowairflow-2.xairflow-webserver

error while importing DAG file in Airflow 2.5.0

when I start my airflow schedular and webserver my bigdata.py file not getting imported, below is the error which am getting.

​Broken DAG: [/home/adminn/airflow/dags/bigdata.py] Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'airflow.providers.apache'

this is the DAG I have written, am I missing something?

here I am trying to pull MySQL table using sqoop and load in HDFS, and scheduling this operation using Airflow.

from airflow.models import DAG
from airflow.contrib.operators.sqoop_operator import SqoopOperator
from airflow.utils.dates import days_ago


Dag_Sqoop_Import = DAG(dag_id="SqoopImport",
                      schedule_interval="* * * * *",
                      start_date=days_ago(2))

sqoop_mysql_import = SqoopOperator(conn_id="sqoop_local",
                                  table="shipmethod",
                                  cmd_type="import",
                                  target_dir="/airflow_sqoopImport",
                                  num_mappers=1,
                                  task_id="SQOOP_Import",
                                  dag=Dag_Sqoop_Import)

sqoop_mysql_import

replies appreciated,thanks.

Solution

  • From airflow 2.x the providers are no longer included by default, but you have to install and then import them and the import path is changed

    In your case you have to:

    Here is the full list of the providers available:

    https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html

    And this is the provider you are looking for:

    https://airflow.apache.org/docs/apache-airflow-providers/packages-ref.html#apache-airflow-providers-apache-sqoop