amazon-web-servicesterraformamazon-cloudwatchterraform-provider-awsamazon-kinesis-firehose

Terraform : Cloudwatch Logs are not being created for my kinesis firehose


I have correctly defined the CloudWatch Log Group and Log Stream. But when I checked in the CloudWatch I found that logs are not being created for my Kinesis Firehose as expected.

Below is my Terraform code. Does anybody know what's wrong here?

resource "aws_cloudwatch_log_group" "AC_firehose_stream_logging_group" {
  name = "/aws/kinesisfirehose/${var.firehose_name}"
  retention_in_days = 30
}

resource "aws_cloudwatch_log_stream" "AC_firehose_stream_logging_stream" {
  log_group_name = aws_cloudwatch_log_group.AC_firehose_stream_logging_group.name
  name           = "ACS3Delivery"
}

resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
  name        = var.firehose_name
  destination = "extended_s3"

  extended_s3_configuration {
    role_arn            = aws_iam_role.sh_firehose_role.arn
    bucket_arn          = var.S3_bucket_arn
    buffer_size         = "128"  # MB
    buffer_interval     = "60" # Seconds (minimum 60)
    prefix              = "${var.s3_raw_ingestion_path}!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/"
    error_output_prefix = "${var.s3_raw_ingestion_path}firehose-error/!{firehose:error-output-type}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/"

    data_format_conversion_configuration { # this part convert the JSON in parquet format, put it in comment if you want only JSON
      input_format_configuration {
        deserializer {
          open_x_json_ser_de {}
        }
      }
      output_format_configuration {
        serializer {
          parquet_ser_de {
            compression = "SNAPPY"
          }
        }
      }
      schema_configuration {
        role_arn      = aws_iam_role.sh_firehose_role.arn
        database_name = var.glue_db_name
        table_name    = var.glue_table_name
      }
      enabled = true
    }
    cloudwatch_logging_options {
      enabled         = true
      log_group_name  = aws_cloudwatch_log_group.AC_firehose_stream_logging_group.name
      log_stream_name = aws_cloudwatch_log_stream.AC_firehose_stream_logging_stream.name
    }

  }
}


Solution

  • I have identified the issue that was preventing CloudWatch logs from being created for my Kinesis Firehose. It appears that the IAM role I had created did not have sufficient permissions to access the S3 bucket where the error logs were being saved. As a result, the IAM role was unable to access the S3 bucket and retrieve the logs. I have now granted the necessary permissions to the IAM role, which should resolve the issue.