Our team is looking to install a custom package we set up internally using a custom pypi server on s3 like this example. We then want to add it to a MWAA requirments.txt file. We configured the IAM policy for MWAA with full bucket access and bucket policy so that MWAA could access the static website to install the whl file. I also tested this locally with the MWAA docker container provided for testing. This worked correctly.
The issue comes when trying to try this on MWAA. I updated the requirments.txt with --extra-index-url www.s3-webite.com --trustedhost s3-url python-package-example==1.56
For some reason this breaks all the dags. When I check cloudwatch, the requirements log shows everything except for python-package-example is installed. ERROR: Could not find a version that satisfies the requirement python-package-example===1.56 (from versions: none)
I am fairly certain it is a networking issue. Has anyone ran into this when trying to add a custom package to your MWAA instance? ERROR: No matching distribution found for python-package-example===1.56
Example bucket policy
{
"Version": "Date",
"Id": "MWAApolicyforcustomS3",
"Statement": [
{
"Sid": "AllowSpecificMWAAAccess",
"Effect": "Allow",
"Principal": {
"AWS": "mwaa-role-example"
},
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::bucket",
"arn:aws:s3:::bucket/*"
]
}
So the issue I realized was that because it was a private subnet the way MWAA was installing pip packages so to fix the issue all I had to do was configure the bucket policy to allow access from the Nat gateway IP.