I am trying to run EMR Serverless job and upload logs into S3 with following logs configuration
--configuration-overrides '{
"monitoringConfiguration": {
"s3MonitoringConfiguration": {
"logUri": "s3://mgosk-temp/emr-serverless-spark/",
"encryptionKeyArn" : "arn:aws:kms:eu-north-1:0000000:key/abcd"
}
}
}'
I am getting following error.
Unable to push logs, please ensure logging destination is valid and execution role has sufficient permissions. Error: "Failed to upload job metadata to mgosk-temp/emr-serverless-spark/applications/.../jobs/.../job-metadata.log: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied".
The execution role have access rights to S3 and KMS assigned , but still looks like the issue with IAM permissions. Which role is used for logs upload execution role or
service-linked roles for EMR Serverless ?
As of now, EMR Serverless doesn't encrypt the job-metadata.log file even though encryptionKeyArn is specified, meaning the headers (eg. s3:x-amz-server-side-encryption) aren't specified. This can therefore cause AccessDenied issue for this file if bucket policy or Organization policy (SCP) have Deny statement based on S3 encryption headers. A workaround until EMR Serverless addresses it, would be to not restrict s3:PutObject request based on encryption headers.