pythonamazon-web-servicesairflowmwaa

Restrict dags in MWAA instance in consumer account via assume role

I have dags under the below locations in the primary root account:

s3://input-read/dags/domainA/*.py  
s3://input-read/dags/domainB/*.py

And dags location passed in the MWAA instance is s3://input-read/dags

When I open the Airflow UI from the Primary account it shows the dags from both the folders which is correct.

Also, I have an IAM user in the secondary account to which I have granted access to only domainA folder and underlying objects via assume role which works perfectly.

I have also granted access to the MWAA instance of the primary account to that IAM user. But here the problem is when I open the Airflow UI in the secondary account's IAM user I still see all the dags which shouldn't happen because on the object level I have restricted access to other folders. So that IAM user should be only able to see dags from domainA folder.

Below is the policy block for Airflow access for the IAM user from primary account:

{
            "Sid": "AllowAirflow",
            "Effect": "Allow",
            "Action": [
                "airflow:ListEnvironments",
                "airflow:GetEnvironment",
                "airflow:ListTagsForResource"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "airflow:CreateWebLoginToken",
            "Resource": [
                "arn:aws:airflow:us-east-1:[account-id]:role/MyAirflowEnvironment/User"
            ]
        }

I also created a role in Airflow UI where you can see the details in the screenshots below:

enter image description here

enter image description here

But I still see all the dags in the IAM user account.

So how can I resolve my problem? Any hints would also help. TIA.

Solution

  • Step by Step Guide towards resolving this problem:

    1. First of all dags path in the MWAA instance has to be simple s3://bucket-name/dags. It will list down all the dags inside the subfolders automatically. No pattern worked as of now. You can have multiple subfolders under dags like domainA, domainB, and so on but you can't provide any wildcards in the path. Doesn't reflect anything.
    2. Even though you have restricted the folder path access and shared the MWAA instance with the IAM user, its Airflow UI will still list down all the dags even though the specific user can't view the objects in s3 via the assumed role console link. Not sure why restricting at the object level still requires an additional step from the Airflow perspective. Not sure if there is any better approach.
    3. Go to the Primary account > Airflow UI -> List Roles -> Modified User role and remove the permission where it can read on all the dags. Specify the custom permissions as shown in my first screenshot for the User role. You can create a custom role as well but you should know what all permissions would be required because if you put insufficient permissions then the IAM user won't be able to open the Airflow UI. Since I wasn't sure on the permissions side, I just chose to modify the User role and remove reading access on all dags and attached that role to the assumed-role user under the List users section.

    If you have multiple users then the best way would be to create multiple roles and attach them to the specific assumed-role-user.

    I am not sure if there is any better approach that exists for such a use case.