Custom Policy and Permissions Set not Working

We're using IAM Identity Center with SSO authentication and permissions sets. I'd created a simple, custom IAM policy that granted some S3 permissions, added that to a permissions set and then applied that to a group and added that group to an account. It all worked.

Then, yesterday, it stopped working. Users in that group no longer had the right permissions. Looking in identity center it said the account to which it was applied had a permissions set status of "outdated" (see screen shot).

I clicked the "update" button in the console but that failed with an error (can no longer recall the exact error message I'm afraid).

So I deleted the policy and then tried to re-add it (via Pulumi). Doing that I now get errors when I try to add it to the group/project via a Pulumi CustomerManagedPolicyAttachment. The error is:

Received a 404 status error: Not supported policy arn:aws:iam::838965223641:policy/cloud_devs_s3_list_all_buckets_policy

I've tried slimming down the policy so it's just:

Policy(
        resource_name,
        name=name,
        path='/',
        description=description,
        policy=json.dumps(
            {
                'Version': '2012-10-17',
                'Statement': [
                    {
                        'Effect': 'Allow',
                        'Action': [
                            's3:PutObject',
                        ],
                        'Resource': 'arn:aws:s3:::*',
                    }
                ],
            }

        ),
    )

But that didn't help. I've seen a few online posts (e.g. this one) that say the policy needs to be "in the account" to which you will apply it but I don't see anyway to put a policy into an account, at least not in Pulumi.

Any idea what I'm missing?

I think I figured it out. Pulumi uses a default "provider" which it derives from the credentials of the current user. If that user is an admin/billing user it'll use that and thus create resources, like policies, globally. But (undocumented) you have to create the policy you want to apply to an identity center group in the same account where you will eventually bind the permission set. The fix is to manually create a Pulumi provider assocaited with the account you will bind to and then things work. Something like:

aws.Provider(
        name,
        allowed_account_ids=[acct_id],
        assume_role=aws.ProviderAssumeRoleArgs(
            role_arn=pulumi.Output.format('arn:aws:iam::{}:role/{}', acct_id, acct_role_name)
        ),
    )

and then use it like:

SomeResource(
  'resouce_name',
  ...,
  opts=ResourceOptions(provider=provider),
)