amazon-s3terraform-provider-awsamazon-kinesisamazon-kinesis-firehose

Terraform : In firehose streams with s3 as the destination, not able to find New line delimiter enabling option


Terraform : In the firehose stream with S3 as the destination, what is the terraform equivalent parameter for enabling "New line delimiter". FYI, There is no dynamic partition, no lambda. The firehose gets the data from kinesis stream which is configured to dynamodb. enter image description here

Please help.


Solution

  • A solution for this was provided in Cloudformation on AWS Repost by Matthew-Gu.

    "For JSON this will look like:

    { "Type": "AppendDelimiterToRecord", "Parameters": [ { "ParameterName": "Delimiter", "ParameterValue": "\n" } ]"

    For Terraform in HCL, this would be:

    resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
      name        = "terraform-kinesis-firehose-extended-s3-test-stream"
      destination = "extended_s3"
    
      extended_s3_configuration {
        role_arn   = aws_iam_role.firehose_role.arn
        bucket_arn = aws_s3_bucket.bucket.arn
    
        processing_configuration {
          enabled = "true"
    
          processors {
            type = "AppendDelimiterToRecord"
    
            parameters {
              parameter_name  = "Delimiter"
              parameter_value = "\\n"
            }
          }
        }
      }
    }
    

    sources: