I have correctly defined the CloudWatch Log Group and Log Stream. But when I checked in the CloudWatch I found that logs are not being created for my Kinesis Firehose as expected.
Below is my Terraform code. Does anybody know what's wrong here?
resource "aws_cloudwatch_log_group" "AC_firehose_stream_logging_group" {
name = "/aws/kinesisfirehose/${var.firehose_name}"
retention_in_days = 30
}
resource "aws_cloudwatch_log_stream" "AC_firehose_stream_logging_stream" {
log_group_name = aws_cloudwatch_log_group.AC_firehose_stream_logging_group.name
name = "ACS3Delivery"
}
resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
name = var.firehose_name
destination = "extended_s3"
extended_s3_configuration {
role_arn = aws_iam_role.sh_firehose_role.arn
bucket_arn = var.S3_bucket_arn
buffer_size = "128" # MB
buffer_interval = "60" # Seconds (minimum 60)
prefix = "${var.s3_raw_ingestion_path}!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/"
error_output_prefix = "${var.s3_raw_ingestion_path}firehose-error/!{firehose:error-output-type}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/"
data_format_conversion_configuration { # this part convert the JSON in parquet format, put it in comment if you want only JSON
input_format_configuration {
deserializer {
open_x_json_ser_de {}
}
}
output_format_configuration {
serializer {
parquet_ser_de {
compression = "SNAPPY"
}
}
}
schema_configuration {
role_arn = aws_iam_role.sh_firehose_role.arn
database_name = var.glue_db_name
table_name = var.glue_table_name
}
enabled = true
}
cloudwatch_logging_options {
enabled = true
log_group_name = aws_cloudwatch_log_group.AC_firehose_stream_logging_group.name
log_stream_name = aws_cloudwatch_log_stream.AC_firehose_stream_logging_stream.name
}
}
}
I have identified the issue that was preventing CloudWatch logs from being created for my Kinesis Firehose. It appears that the IAM role I had created did not have sufficient permissions to access the S3 bucket where the error logs were being saved. As a result, the IAM role was unable to access the S3 bucket and retrieve the logs. I have now granted the necessary permissions to the IAM role, which should resolve the issue.