I'm trying to use both a wheel dependency that I created and, additionally, use PyPi dependencies in MWAA.
I was basing the steps I did on this article: https://repost.aws/knowledge-center/mwaa-install-custom-packages
My requirements.txt file:
--find-links /usr/local/airflow/plugins/plugins
--no-index
apache-airflow-providers-snowflake==4.0.2
apache-airflow-providers-mysql==4.0.0
awswrangler==2.19.0
sagemaker==2.140.1
/usr/local/airflow/plugins/plugins/example_wheel.whl
in the plugins dir I created a plugins.zip file with the wheel inside of it at it's root.
When running Airflow locally using this project: https://github.com/aws/aws-mwaa-local-runner Both the PyPi dependencies are not recognized and the wheel dependency isn't recognized.
I see the following error in the UI if the awswrangler import is first in the code:
ModuleNotFoundError: No module named 'awswrangler'
and the same error for import a class defined in my wheel if that import comes first in the code.
If I keep only the PyPi imports in the requirements file and remove the --index and --find-links and the wheel import then at least these imports are not failing.
What am I doing wrong?
Found the answer, I can use both local wheels and PyPi dependencies. The following requirements file worked:
awswrangler==2.19.0
sagemaker==2.140.1
/usr/local/airflow/plugins/plugins/example_wheel.whl
Also, I tested the contents of the plugins folder and found that even though I place the wheel in the root of the zip file, the actual path of the wheel is as I wrote in the requirements above.