I am trying to read up on AWS' MWAA service. We run a moderately small number of ETL processes, but we are looking to expand in the near future, and I am trying to get out of having to manage our current Airflow production environment.
The question I have is simple: has anyone been able to install Python dependencies via requirements.txt that are hosted in private repos on say GitHub? I am guessing there would have to be a way to pass in an authentication token to the requirements file somehow.
An immediate work around (perhaps ugly) is to make copies of the repos somewhere in AWS? Maybe S3?
I appreciate any feedback!
The suggested way is to build the private repo as a whl, then place that whl in the Airflow MWAA plugins.zip.