Amazon Kinesis Data Application throws "not authorized to perform: cloudwatch:PutMetricData" error

I have an AWS Kinesis data application running Apache Flink 1.13 project. I have provided the following access in the policy attached to the IAM role. I have verified that the IAM role has this policy attached in the AWS console.

AWS Role for Kinesis application (in Terraform):

resource "aws_iam_role" "kinesis_application_role" {
  name = "dev-kinesis_application_role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          "Service" : "kinesisanalytics.amazonaws.com"
        }
      },
    ]
  })

Policy allowing cloudwatch:PutMetricData action.

    {
     "Action": [
       "cloudwatch:PutMetricData"
     ],
     "Effect": "Allow",
     "Resource": [
         "arn:aws:cloudwatch:us-east-1:<aws_account_id>:namespace/*",
         "arn:aws:cloudwatch:us-east-1:<aws_account_id>:metric-stream/*", 
         "arn:aws:cloudwatch:us-east-1:<aws_account_id>:alarm:*",
         "arn:aws:cloudwatch:us-east-1:<aws_account_id>:insight-rule/*",
         "arn:aws:cloudwatch:us-east-1:<aws_account_id>:dashboard/*"
],
     "Sid": "Cloudwatch"
    }

However I get the error not authorized to perform: cloudwatch:PutMetricData in the Cloudwatch logs. What other access is needed for this action?

    {
        
        "message": "[2023-04-05 20:17:23.165505] [0x000002d4]
    [0x00007fa9e42f7700] [error] [metrics_manager.cc:155] 
    Metrics upload failed. | Code: AccessDenied | Message: 
    User: arn:aws:sts::008819713222:assumed-role/dev-
    kinesis_application_role/kiam-kiam is not authorized to perform: 
    cloudwatch:PutMetricData because no identity-based policy allows 
the cloudwatch:PutMetricData action with address : 52.46.143.92 | 
Request was: Action=PutMetricData&Namespace=KinesisProducerLibrary&
MetricData.member.1.MetricName=UserRecordsPending&
MetricData.member.1.Timestamp=2023-04-05T20%3A06%3A54Z&
    MetricData.member.1.StatisticValues.SampleCount=299&
    MetricData.member.1.StatisticValues.Sum=0&
    MetricData.member.1.StatisticValues.Minimum=0&
    MetricData.member.1.StatisticValues.Maximum=0&
    MetricData.member.1.Unit=Count&MetricData.member.2.MetricName=UserRecordsPending&
    MetricData.member.2.Dimensions.member.1.Name=StreamName&
    MetricData.member.2.Dimensions.member.1.Value=Events-Dev&
    MetricData.member.2.Timestamp=2023-04-05T20%3A06%3A54Z&
    MetricData.member.2.StatisticValues.SampleCount=299&
    MetricData.member.2.StatisticValues.Sum=0&
    MetricData.member.2.StatisticValues.Minimum=0&
    MetricData.member.2.StatisticValues.Maximum=0&
    MetricData.member.2.Unit=Count&Version=2010-08-01",
    
        "messageSchemaVersion": "1",
        "messageType": "ERROR",
        "threadName": "kpl-daemon-0003"
    }

and warning:

    [2023-04-05 17:51:13.897017] [0x000002d4][0x00007fa9e10f2700] [warning] [AWS Log: WARN](AWSErrorMarshaller)
Encountered AWSError 'AccessDenied': User: arn:aws:sts::008819713222:assumed-role/dev-kinesis_application_role/kiam-kiam 
is not authorized to perform: cloudwatch:PutMetricData 
because no identity-based policy allows the 
cloudwatch:PutMetricData action

Solution

Unfortunately, you can't do better than "Resource":"*" for cloudwatch:PutMetricsData.

As per Amazon's documentation, cloudwatch:PutMetricsData doesn't support resource constraints. Scroll down to "PutMetricsData" in the "Actions defined by Amazon CloudWatch" table and look at the "Resource types" column: it is blank, which indicates that that "If there is no value for this column, you must specify all resources ("*") to which the policy applies in the Resource element of your policy statement.".

Good job at trying to lock down your policies as much as possible, though!