Configure IAM for Fluent-bit on AWS EKS

I'm trying to configure the official helm Fluent-Bit for my EKS to send apps logs to AWS CloudWatch.

I configure everything with Terraform. Here's my file

resource "kubernetes_namespace" "fluentbit" {
  metadata {
    name = var.fluentbit_namespace
  }
}

resource "kubernetes_service_account" "fluentbit" {
  metadata {
    namespace = var.fluentbit_namespace
    name      = var.fluentbit_service_account_name
    annotations = {
      "eks.amazonaws.com/role-arn" = aws_iam_role.cloudwatch_logs_role.arn
    }
  }
}

resource "helm_release" "fluent_bit_daemonset" {
  repository = "https://fluent.github.io/helm-charts"
  chart      = "fluent-bit"
  version    = "0.15.15"

  name             = "fluent-bit"
  namespace        = var.fluentbit_namespace
  create_namespace = false
  cleanup_on_fail  = true
  values = [
    templatefile("${path.cwd}/templates/fluentbit_config_template.yaml", {
      service_account_name   = var.fluentbit_service_account_name,
      create_service_account = false
      image_version          = var.fluentbit_image_version,
      region                 = var.cluster_region
      log_group_name         = var.fluentbit_log_group_name
      log_stream_name        = var.fluentbit_log_stream_name
      role_arn               = aws_iam_role.cloudwatch_logs_role.arn
      cluster_name           = "xxxx-kubernetes-sandbox"
    }),
  ]
}

resource "aws_cloudwatch_log_group" "cloudwatch_log_group" {
  name = var.fluentbit_log_group_name
}


resource "aws_iam_role" "cloudwatch_logs_role" {
  name = "cloudwatch-logs-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = "sts:AssumeRole",
        Effect = "Allow",
        Principal = {
          Service = "logs.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_iam_policy" "cloudwatch_logs_policy" {
  name = "fluentbit-role"

  description = "Role use to create logs from K8S to cloudwatch"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "logs:CreateLogGroup",
          "logs:CreateLogStream",
          "logs:PutLogEvents",
          "logs:DescribeLogGroups",
          "logs:DescribeLogStreams"
        ],
        Effect   = "Allow",
        Resource = "*"
      }
    ]
  })
}

resource "aws_iam_policy_attachment" "attach_cloudwatch_logs_policy" {
  name       = "attach-cloudwatch-logs-policy"
  policy_arn = aws_iam_policy.cloudwatch_logs_policy.arn
  roles      = [aws_iam_role.cloudwatch_logs_role.name]
}

The fluentbit configuration, which is not complicated for the moment

image:
   tag: ${image_version}
 
serviceAccount:
   name: ${service_account_name}
   create: ${create_service_account}

config:
  service: |
    [SERVICE]
        Flush        2
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    {{ .Values.service.port }}

  inputs: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10    

  filters: |
    [FILTER]
        Name                kubernetes
        Match               application.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     application.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Labels              Off
        Annotations         Off
        Use_Kubelet         On
        Kubelet_Port        10250
        Buffer_Size         0

  outputs: |
    [OUTPUT]
        Name                cloudwatch_logs
        Match               *
        region              ${region}
        log_group_name      ${log_group_name}
        log_stream_name     ${log_stream_name}
        auto_create_group   false
        sts_endpoint        https://sts.${region}.amazonaws.com

I end-up with error

CreateLogStream API responded with error='AccessDeniedException'  
Failed to create log stream
Failed to send events

If I add the option role_arn in the fluentbit configuration file the error is different

[2023/11/07 15:16:32] [error] [signv4] Provider returned no credentials, service=logs
[2023/11/07 15:16:32] [error] [aws_client] could not sign request
[2023/11/07 15:16:32] [error] [aws_credentials] STS assume role request failed 
[2023/11/07 15:16:32] [ warn] [aws_credentials] No cached credentials are available and a credential refresh is already in progress. The current co-routine will retry.

I try to use the module https://registry.terraform.io/modules/terraform-aws-modules/iam/aws/latest/submodules/iam-role-for-service-accounts-eks on the service account but it doesn't change anything

I don't know if there's no another problem into my cluster configuration ? Do you have any hint ? Thanks.

Resolved by

1. use the module "IRSA for K8S". https://registry.terraform.io/modules/terraform-aws-modules/iam/aws/latest/submodules/iam-role-for-service-accounts-eks

module "fluentbit_cloudwatch_irsa_role" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "5.11.1"

  role_name = "company-k8s-fluentbit-cloudwatch-irsa-role-${terraform.workspace}"

  oidc_providers = {
    main = {
      provider_arn               = data.terraform_remote_state.k8s-cluster.outputs.cluster_oidc_provider_arn
      namespace_service_accounts = ["${var.fluentbit_namespace}:${var.fluentbit_service_account_name}"]
    }
  }
}

2. Fix the permissions (need to restrict to logs:* for better security)

resource "aws_iam_policy" "cloudwatch_logs_policy" {
  name = "fluentbit-role"

  description = "Role use to create logs from K8S to cloudwatch"

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [
      {
        Action = [
          "ec2:DescribeTags",
          "logs:PutLogEvents",
          "cloudwatch:PutMetricData",
          "logs:DescribeLogStreams",
          "logs:DescribeLogGroups",
          "logs:CreateLogStream",
          "logs:CreateLogGroup"
        ],
        Effect   = "Allow",
        Resource = "*"
      }
    ]
  })
}