pythonairflowairflow-2.xmwaaairflow-webserver

All DAGs broken after MWAA update from 2.0.2 to 2.2.2

I am getting the following errors in AWS MWAA UI after I updated from 2.0.2 to 2.2.2 I have exhaustively searched for more details on these errors to no avail

from airflow.providers.slack.operators.slack_webhook import SlackWebhookOperator
ModuleNotFoundError: No module named 'airflow.providers.slack'

Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/.___init__.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/operators/._aws_glue_crawler_operator.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/operators/._aws_glue_job_operator.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/operators/.___init__.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/hooks/.___init__.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/sensors/._aws_glue_job_sensor.py] source code string cannot contain null bytes
Broken plugin: [/usr/local/airflow/plugins/__MACOSX/awsairflowlib/sensors/.___init__.py] source code string cannot contain null bytes

The CloudWatch logs from the WebServer are no helpful either:

ValueError: source code string cannot contain null bytes

PFA enter image description here

My plugins zip file while I was using 2.0.2 was of the following structure and it worked just fine. I believe this is not working for v2.2.2:

s3://{my-bucket-name}
└── DAGS
    ├── dags 
    │   └── dag1.py  
    └── plugins
        ├── __init__.py
        └── operators
        |   ├── __init__.py
        |   └── aws_glue_operator.py
        |___ sensors
             |__aws_glue_job_sensor.py

enter image description here

Here is my requirements.txt contents

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt"
apache-airflow-providers-postgres[amazon]
apache-airflow-providers-cncf-kubernetes
psycopg2-binary==2.8.6
fsspec
s3fs
pandas
sagemaker==v1.72
dag-factory==0.7.2
requests
boto3
botocore
apache-airflow-providers-amazon
PyOpenSSL
oauth2client
gspread

Any help is appreciated

Solution

  • With MWAA 2.2.2 the plugins zip file and the requirements.txt need to be downloaded from here Any version conflicts in requirements.txt and Airflow should be cross-checked from here

    With these two files I was able to fix the broken DAGs in 2.2.2