dockerairflowairflow-2.x

Airflow import error after updating from version 2.0.2 to 2.8.1

I have a local dev environment where I am trying to update airflow version from 2.0.2 to 2.8.1. But I am getting error as below: If I go back to 2.0.2 things works as expected. Does anyone has some idea what is going on here.

local-runner-1  | The above exception was the direct cause of the following exception:
local-runner-1  | 
local-runner-1  | Traceback (most recent call last):
local-runner-1  |   File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/models/dagbag.py", line 344, in parse
local-runner-1  |     loader.exec_module(new_module)
local-runner-1  |   File "<frozen importlib._bootstrap_external>", line 940, in exec_module
local-runner-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
local-runner-1  |   File "/usr/local/airflow/dags/my-dag-folder/other_folder/dag_data_export.py", line 5, in <module>
local-runner-1  |     from common.create_task import create_task
local-runner-1  |   File "/usr/local/airflow/plugins/common/create_task.py", line 1, in <module>
local-runner-1  |     from common.my_ssh_operator import YmSSHOperator
local-runner-1  |   File "/usr/local/airflow/plugins/common/my_ssh_operator.py", line 4, in <module>
local-runner-1  |     from airflow.contrib.operators.ssh_operator import SSHOperator
local-runner-1  |   File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/deprecation_tools.py", line 63, in getattr_with_deprecation
local-runner-1  |     raise ImportError(error_message) from e
local-runner-1  | ImportError: Could not import `airflow.providers.ssh.operators.ssh.SSHOperator` while trying to import `airflow.contrib.operators.ssh_operator.SSHOperator`.
local-runner-1  | ERROR [airflow.models.dagbag.DagBag] Failed to import: /usr/local/airflow/dags/my-dag-folder/other_folder/dag_data_rdbms_export.py
local-runner-1  | Traceback (most recent call last):
local-runner-1  |   File "/usr/local/airflow/.local/lib/python3.11/site-packages/airflow/utils/deprecation_tools.py", line 56, in getattr_with_deprecation
local-runner-1  |     return getattr(importlib.import_module(new_module), new_class_name)
local-runner-1  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
local-runner-1  |   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
local-runner-1  |     return _bootstrap._gcd_import(name[level:], package, level)
local-runner-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
local-runner-1  |   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
local-runner-1  |   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
local-runner-1  |   File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
local-runner-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
local-runner-1  |   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
local-runner-1  |   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
local-runner-1  |   File "<frozen importlib._bootstrap>", line 1126, in _find_and_load_unlocked
local-runner-1  |   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
local-runner-1  |   File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
local-runner-1  |   File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
local-runner-1  |   File "<frozen importlib._bootstrap>", line 1140, in _find_and_load_unlocked
Solution

  • The traceback indicates that there's an ImportError occurring while trying to import airflow.contrib.operators.ssh_operator.SSHOperator. The specific error message states:

    ImportError: Could not import `airflow.providers.ssh.operators.ssh.SSHOperator` while trying to import `airflow.contrib.operators.ssh_operator.SSHOperator`.

    Transitioning from version 2.0.2 to 2.8.1 signifies a significant leap, indicating numerous alterations. It's essential to review the change log of providers and ascertain what components are included by default in the MWAA environment. Referencing the MWAA documentation for package listings is crucial in this scenario. https://docs.aws.amazon.com/mwaa/latest/userguide/connections-packages.html#connections-packages-table-281

    Specifically, in version 2.8.1, it's notable that the environment no longer incorporates the SSH Connection feature, a contrast to its presence in version 2.0.2.

    During the upgrade process, it's common to encounter changes such as: