I have a Nodejs server running Amazon Linux 2023 that is acting as an event subscriber. I would like to configure some monitoring on it so that I can send logs to my monitoring service. I was able to get the user-data section to run all the other commands successfully except the line that writes the config file. Once this is working, I will need to expand it to install and configure Prometheus and node-exporter as well.
I'm not sure but I think I want it to run during the runcmd section so I can create the config file and then start the service based on the config file I just created.
If I remove the config file, everything else is successful. I've tried using the write_file section but don't think that will work since I will need to enable and start the services once the configuration files are written. I've tried every combination of quotes with echo, cat, and printf, I can think of. Started with the cloud_config_config heredoc directly in the main.tf file and was trying to just pull in the filebeat_config. i.e.
main.tf file
...
user_data = <<-EOF
#cloud-config
package_update: true
package_upgrade: true
selinux:
mode: enforcing
runcmd:
...
- echo "${local.filebeat_config}" > /data_vol/configs/filebeats/filebeat.yml
EOF
Currently getting this error: (user_data value has been moved to locals.tf see below)
2023-08-17 00:19:11,336 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 52 column 2: "while scanning a block scalar
in "<unicode string>", line 52, column 2:
> /data_vol/configs/filebeats/fi ...
^
expected a comment or a line break, but found '/'
in "<unicode string>", line 52, column 4:
> /data_vol/configs/filebeats/file ...
^"
2023-08-17 00:19:11,654 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 52 column 2: "while scanning a block scalar
in "<unicode string>", line 52, column 2:
> /data_vol/configs/filebeats/fi ...
^
expected a comment or a line break, but found '/'
in "<unicode string>", line 52, column 4:
> /data_vol/configs/filebeats/file ...
^"
2023-08-17 00:19:11,667 - cloud_config.py[WARNING]: Failed at merging in cloud config part from part-001: empty cloud config
main.tf file
...
resource "aws_instance" "event_listener" {
ami = data.aws_ami.amzLinux.id
associate_public_ip_address = false
instance_type = var.instance_type
key_name = data.aws_key_pair.existing_key.key_name
subnet_id = data.aws_subnets.private.ids[1]
user_data_replace_on_change = true
user_data = local.cloud_config_config
...
}
...
locals.tf file
locals {
...
cloud_config_config = <<-EOF
#cloud-config
package_update: true
package_upgrade: true
selinux:
mode: enforcing
runcmd:
- yum install -y https://s3.${var.aws_region}.amazonaws.com/amazon-ssm-${var.aws_region}/latest/linux_amd64/amazon-ssm-agent.rpm
- curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.9.0-x86_64.rpm
- rpm -vi filebeat-8.9.0-x86_64.rpm
- rm -f filebeat-8.9.0-x86_64.rpm
- mkdir /data_vol -p
- mkfs -t xfs /dev/nvme1n1
- mount /dev/nvme1n1 /data_vol
- echo "/dev/nvme1n1 /data_vol xfs defaults,nofail 0 2" >> /etc/fstab
- mkdir -p /usr/share/ca-certificates/coralogix
- curl -o /usr/share/ca-certificates/coralogix/ca.crt https://coralogix-public.s3-us-east-2.amazonaws.com/certificate/ca.crt
- dnf update -y && dnf install -y nano git nodejs
- npm install -g pm2
- mkdir -p /data_vol/configs/filebeats /data_vol/configs/prometheus /data_vol/configs/node-exporter /data_vol/logs/sf_event_listener
- cat << ${local.filebeat_config} > /data_vol/configs/filebeats/filebeat.yml
EOF
filebeat_config = <<-EOT
# ============================== Filebeat inputs ===============================
filebeat.inputs:
- type: log
id: sf_event_listener_id
paths:
- "/data_vol/logs/sf_event_listener/err.log"
fields_under_root: true
fields:
PRIVATE_KEY: "${var.coralogix_auth.private_key}"
COMPANY_ID: ${var.coralogix_auth.company_id}
APP_NAME: "salesforce"
SUB_SYSTEM: "system_event"
multiline.type: pattern
multiline.pattern: '^\d{4}-\d{2}-\d{2}'
multiline.negate: true
multiline.match: after
# ============================== Filebeat modules ==============================
filebeat.config.modules:
path: $${path.config}/modules.d/*.yml
reload.enabled: false
output.logstash:
enabled: true
hosts: ["${var.coralogix_auth.hosts[0]}"]
tls.certificate_authorities: ["/usr/share/ca-certificates/coralogix/ca.crt"]
ssl.certificate_authorities: ["/usr/share/ca-certificates/coralogix/ca.crt"]
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
EOT
}
Any suggestions would be appreciated.
The direct problem here is that you are trying to construct a YAML document by concatenating strings together and it's inserting something that's making the resulting document invalid YAML syntax. You can avoid this class of problems by generating YAML using yamlencode instead, so that Terraform is the one responsible for producing valid YAML syntax and you only need to worry about making sure the data structure is correctly-shaped for cloud-init to decode.
locals {
cloud_config_config = <<-EOT
#cloud-config
${yamlencode({
package_update = true
package_upgrade = true
selinux = {
mode = "enforcing"
}
runcmd = [
"yum install -y https://s3.${var.aws_region}.amazonaws.com/amazon-ssm-${var.aws_region}/latest/linux_amd64/amazon-ssm-agent.rpm",
# (etc etc)
]
})}
EOT
}
Terraform will generate a valid YAML document containing the values you specify, using appropriate formatting/escaping to deal with problems such as there being newline characters in your local.filebeat_config value.
Although it's far outside the scope of what you were asking about, you might be interested to know that cloud-init has built-in support for writing files to disk, partitioning and formatting disks, mounting partitions into the virtual filesystem, and installing RPM packages, and so you might be able to replace this big sequence of imperative commands with a declarative configuration using cloud-init's other modules instead.
That sort of approach is often a better blend with Terraform's declarative style, though of course there's no harm in doing it imperatively via shell commands if you want to.