bashyamlcloud-init

Using valid YAML in cloud-init to run commands that add text to file

Using cloud-init, how can I run add text (list of strings) to a file? Also, do I need to escape the colon : character? It seems that that problem is YAML validation but could not find any examples to help me out.

Here is what I've tried. None of the echo commands seem to be valid.

#cloud-config

runcmd:
  - aws s3 cp s3://my-bucket/elasticsearch/elasticsearch.tar.gz /opt/elastic/elasticsearch.tar.gz
  - tar -xf /opt/elastic/elasticsearch.tar.gz
  - ln -sv /opt/elastic/elasticsearch-7.7.6 /opt/elastic/elasticsearch
  - cp /opt/elastic/elasticsearch/config/elasticsearch.yml /opt/elastic/elasticsearch/config/elasticsearch.yml.bkp
  - echo 'cluster.name: DEMO' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - echo 'node.name: node1' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - echo 'path.data: /opt/elastic/data' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - echo 'path.logs: /opt/elastic/logs' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - echo 'network.host: host1.domain.internal' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - echo 'http.port: 9200' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - echo 'discovery.seed_hosts: ["host1.domain.internal", "host2.domain.internal", "host3.domain.internal"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - echo 'cluster.initial_master_nodes: ["node1", "node2", "node3"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
Solution

  • You're insufficiently quoting the values in your list, so the : in your echo statements is getting interpreted as the key: value separator. You want to quote the entire contents of each line, and the best way of doing this is probably using one of the YAML quote operators (>, the folding quote operator, or | the literal quote operator). You'll find some documentation on this topic here.

    Like this:

    #cloud-config

runcmd:
  - |
    aws s3 cp s3://my-bucket/elasticsearch/elasticsearch.tar.gz /opt/elastic/elasticsearch.tar.gz
  - |
    tar -xf /opt/elastic/elasticsearch.tar.gz
  - |
    ln -sv /opt/elastic/elasticsearch-7.7.6 /opt/elastic/elasticsearch
  - |
    cp /opt/elastic/elasticsearch/config/elasticsearch.yml /opt/elastic/elasticsearch/config/elasticsearch.yml.bkp
  - |
    echo 'cluster.name: DEMO' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'node.name: node1' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'path.data: /opt/elastic/data' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'path.logs: /opt/elastic/logs' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'network.host: host1.domain.internal' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'http.port: 9200' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'discovery.seed_hosts: ["host1.domain.internal", "host2.domain.internal", "host3.domain.internal"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml
  - |
    echo 'cluster.initial_master_nodes: ["node1", "node2", "node3"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml

    You can verify this parses correctly by feeding it through a YAML-to-JSON converter, which will show you:

    [
  "aws s3 cp s3://my-bucket/elasticsearch/elasticsearch.tar.gz /opt/elastic/elasticsearch.tar.gz\n",
  "tar -xf /opt/elastic/elasticsearch.tar.gz\n",
  "ln -sv /opt/elastic/elasticsearch-7.7.6 /opt/elastic/elasticsearch\n",
  "cp /opt/elastic/elasticsearch/config/elasticsearch.yml /opt/elastic/elasticsearch/config/elasticsearch.yml.bkp\n",
  "echo 'cluster.name: DEMO' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'node.name: node1' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'path.data: /opt/elastic/data' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'path.logs: /opt/elastic/logs' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'network.host: host1.domain.internal' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'http.port: 9200' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'discovery.seed_hosts: [\"host1.domain.internal\", \"host2.domain.internal\", \"host3.domain.internal\"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n",
  "echo 'cluster.initial_master_nodes: [\"node1\", \"node2\", \"node3\"]' >> /opt/elastic/elasticsearch/config/elasticsearch.yml\n"
]

    You should be able to combine those lines into a single multi-line shell script, like this (I've taken the liberty of replacing the multiple echo statements with a single "here"-document):

    #cloud-config

runcmd:
  - |
    aws s3 cp s3://my-bucket/elasticsearch/elasticsearch.tar.gz /opt/elastic/elasticsearch.tar.gz
    tar -xf /opt/elastic/elasticsearch.tar.gz
    ln -sv /opt/elastic/elasticsearch-7.7.6 /opt/elastic/elasticsearch
    cp /opt/elastic/elasticsearch/config/elasticsearch.yml /opt/elastic/elasticsearch/config/elasticsearch.yml.bkp

    cat >> /opt/elastic/elasticsearch/config/elasticsearch.yml <<EOF
    cluster.name: DEMO
    node.name: node1
    path.data: /opt/elastic/data
    path.logs: /opt/elastic/logs
    network.host: host1.domain.internal
    http.port: 9200
    discovery.seed_hosts: ["host1.domain.internal", "host2.domain.internal", "host3.domain.internal"]
    cluster.initial_master_nodes: ["node1", "node2", "node3"]
    EOF

    That's probably both easier to read and easier to maintain.