airflowmwaa

Can MWAA requirements.txt be automatically set to the latest version?


We have an automated upload system for our DAG's to MWAA.

Is there a way to have the requirements.txt in the S3 bucket be automatically set to its latest version?

I cannot find an option for this in the AWS console.


Solution

  • It's in the setup of MWAA. Edit MWAA instance, in the requirements.txt field, set the version to the latest.

    Also,

    MWAA uses requirements file to create the container image. So when you upload requirement file and specify it in the edit options and save, images are created(it takes couple of minutes to do it). Ideally image should only be updated when you have new python libraries to add. if you still want to automate, try to use lambda S3 trigger and use mwaa cli to trigger the update-environment command.