How to stream custom logs to cloudwatch from AL2023 instances

We use amazon-elastic-beanstalk to deploy part of our stack. We have just migrated our base platform from PHP 7.4 which uses Amazon Linux 2, to PHP 8.2 which uses Amazon Linux 2023.

Default system logs are being streamed properly (/var/log/nginx/access.log, /var/log/eb-hooks.log, etc.), but custom log streaming to cloudwatch is not working.

I followed this guide https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/AWSHowTo.cloudwatchlogs.html in order to enable log streaming within amazon-linux-2

(The guide has not been updated to amazon-linux-2023)

In AL2 we declare the logs we want to be sent creating an ebextension that added the files within /etc/awslogs/config/logs.conf like described in this example from awsdocs

Also I had to enable Cloudwatch log streaming by creating another ebextension: .ebextension/95-logs-streamtocloudwatch.config

option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
    StreamLogs: true
    DeleteOnTerminate: false
    RetentionInDays: 30

And finally I created a policy and add it to the role defined in the EC2 instance profile like documented in the official guide: Using Elastic Beanstalk with Amazon CloudWatch Logs

{
 "Version": "2012-10-17",
 "Statement": [
 {
   "Effect": "Allow",
   "Action": [
     "logs:CreateLogGroup",
     "logs:CreateLogStream",
     "logs:PutLogEvents",
     "logs:DescribeLogStreams"
   ],
   "Resource": [
   "*"
   ]
 }
 ]
}

from: official source

That configuration worked fine within amazon-linux-2 instances. But is not working within Amazon Linux 2023 instances

I have found the solution thanks to this existing answer

What I have done is to edit the .ebextension file in order to set the configuration for the cloudwatch agent under /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/ folder.

This is my resulting .ebextension file

option_settings:
  aws:elasticbeanstalk:cloudwatch:logs:
    StreamLogs: true
    DeleteOnTerminate: false
    RetentionInDays: 30

packages:
  yum:
    amazon-cloudwatch-agent: []

files:
  "/etc/awslogs/awscli.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [plugins]
      cwlogs = cwlogs
      [default]
      region = `{"Ref":"AWS::Region"}`
  "/etc/awslogs/awslogs.conf" :
    mode: "000600"
    owner: root
    group: root
    content: |
      [general]
      state_file = /var/lib/awslogs/agent-state
  "/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/custom_logs.json":
    mode: "000600"
    owner: root
    group: root
    content: |
      {
        "logs": {
          "logs_collected": {
            "files": {
              "collect_list": [
                {
                  "file_path": "/var/log/php-fpm/mycustomlog.log",
                  "log_group_name": "/aws/elasticbeanstalk/erp-prod-php8/var/log/php-fpm/mycustomlog.log",
                  "log_stream_name": "{instance_id}"
                }
              ]
            }
          }
        }
      }

commands:
  "01":
    command: systemctl enable amazon-cloudwatch-agent.service
  "02":
    command: systemctl restart amazon-cloudwatch-agent