cloud-init fails on Amazon Linux 2023 but works on AmazonLinux2

Here's my very simple test script that I want to run on ec2 launch via cloud-init using terraform.

data "cloudinit_config" "cloudinit" {
  gzip          = false
  base64_encode = false

  part {
    content_type = "text/x-shellscript"
    content      = <<EOT
      # !/bin/bash
      echo "Running User Data on $HOSTNAME"
      EOT
    filename     = "cloudinit.sh"
  }
}

which is used in the ec2 node config as follows: user_data = data.cloudinit_config.cloudinit.rendered

If I create the node with an AmazonLinux2 image, the script runs fine and I can see the echo printed in /var/log/cloud-init-output.log

But the same setup with just the change to AmazonLinux 2023 doesn't run the script. Here's the AMI related config:

data "aws_ami" "amazon-linux-2023" {
  most_recent = true
  owners      = ["amazon"]
  filter {
    name   = "architecture"
    values = ["x86_64"]
  }
  filter {
    name   = "name"
    values = ["al2023-ami-2023.*"]
    # values = ["amzn2-ami-kernel-5.10-hvm*"]
  }
}

Some errors on the bottom of the cloud-init-output.log doesn't make much sense to me. Any clues?

Cloud-init v. 22.2.2 running 'modules:config' at Thu, 04 Jul 2024 23:31:18 +0000. Up 10.32 seconds.
Cloud-init v. 22.2.2 running 'modules:final' at Thu, 04 Jul 2024 23:31:19 +0000. Up 11.22 seconds.
2024-07-04 23:31:19,650 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2024-07-04 23:31:19,650 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3.9/site-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 22.2.2 finished at Thu, 04 Jul 2024 23:31:19 +0000. Datasource DataSourceEc2.  Up 11.77 seconds

I can see that on both amazon linux versions, terraform is attaching the following multipart user data which is identical so clearly nothing different about terraform setup either. I tried some older versions of amazon linux 2023 too but no luck:

Content-Type: multipart/mixed; boundary="MIMEBOUNDARY"
MIME-Version: 1.0

--MIMEBOUNDARY
Content-Disposition: attachment; filename="cloudinit.sh"
Content-Transfer-Encoding: 7bit
Content-Type: text/x-shellscript
Mime-Version: 1.0

      # !/bin/bash
      echo "Running User Data on $HOSTNAME"

--MIMEBOUNDARY--

Tried asking on AWS rePost but didn't get any clues there either so trying good 'ol SO community :)

Tried running the same on AmazonLinux2 and it works fine.

Looks like Amazonlinux:2023 needs the script interpreter shebang which was probably optional for Amazonlinux2. My script had incorrect EOT and an extra space that was effective rendering a missing shebang. Here's the fixed code:

data "cloudinit_config" "cloudinit" {
  gzip          = false
  base64_encode = false

  part {
    content_type = "text/x-shellscript"
    content      = <<-EOT
      #!/bin/bash
      echo "Running User Data on $HOSTNAME"
      EOT
    filename     = "cloudinit.sh"
  }
}