Azure DevOps Pipelines Agent Pool via Terraformed Scale Set

I was tasked with creating a Linux-based Scale Set for use with Azure DevOps Pipelines in Terraform.

I have everything set up for the basics; however, when I click on the Agents tab in the Agent pools area of my DevOps Project, I get the message:

No agents are connected Azure virtual machine scale set agents will appear here when they are created.

I assume that I need the agent installed using these instructions.

What I have done so far:

1. Terraform my Azure Scale Set using azurerm_linux_virtual_machine_scale_set - I am using UbuntuServer 18.04-LTS
2. Add the CustomScript extension via azurerm_virtual_machine_scale_set_extension
3. Pass in a custom commandToExecute parameter read from a file in Terraform
4. In my DevOps project, add a new Agent pool that uses the Scale Set created

In my custom script, I have the basic download and unpacking of the Linux agent:

mkdir azure-agent
curl -sSL https://vstsagentpackage.azureedge.net/agent/2.187.2/vsts-agent-linux-x64-2.187.2.tar.gz --output vsts-agent.tar.gz
tar -zxf vsts-agent.tar.gz -C azure-agent

Per the instructions, I need to run the config.sh script, which is interactive.

My questions are these:

1. Is there a better way to do this than running the custom script to install the agent?
2. If there is not a better way to do this, how can I get the data that the agent needs into the VMs without running the config.sh script?

I looked through the output of az vm extension image list --output table and did not see anything in there besides the Microsoft.Azure.Monitor set of extensions, which does not seem like what I'd want to use and I didn't see any anything labeled "vsts". Using az vm image list --all didn't turn up anything I noticed, either.

Update 1

After playing around I was able to find Microsoft.VisualStudio.Services.TeamServicesAgentLinux v1.21, with some basic settings:

  settings = jsonencode({
    "isPipelinesAgent"        = true
    "agentFolder"             = "/agent"
    "agentDownloadUrl"        = "https://vstsagentpackage.azureedge.net/agent/2.187.2/vsts-agent-linux-x64-2.187.2.tar.gz"
    "enableScriptDownloadUrl" = "https://vstsagenttools.blob.core.windows.net/tools/ElasticPools/Linux/6/enableagent.sh"
  })

This still doesn't give me options for what the config.sh script asks for when run.

Update 2

I followed the suggestion from @Joy and was able to verify that my script I created runs successfully. Which looks like this and functions just fine if I SSH into a VM and execute it:

set -e

sudo apt-get update
sudo apt install -y jq apt-transport-https ca-certificates curl gnupg lsb-release python3 python3-pip

sudo apt install -y liblttng-ust0 libkrb5-3 zlib1g

sudo apt install -y libssl1.1 || \
  sudo apt install -y libssl1.0.2 || \
  sudo apt install -y libssl1.0.0

sudo apt install -y libicu67 || \
  sudo apt install -y libicu66 || \
  sudo apt install -y libicu63 || \
  sudo apt install -y libicu60 || \
  sudo apt install -y libicu57 || \
  sudo apt install -y libicu55 || \
  sudo apt install -y libicu52

# download the Azure script & run the setup
if [[ -d vsts-install ]]; then
  rm -Rf vsts-install
fi

mkdir vsts-install
cd vsts-install

curl -sSL https://vstsagentpackage.azureedge.net/agent/2.187.2/vsts-agent-linux-x64-2.187.2.tar.gz --output vsts.tgz
tar -zxf vsts.tgz

./config.sh --unattended \
            --url https://dev.azure.com/MyOrganization \
            --auth pat \
            --token "MY_TOKEN" \
            --pool TerraformedDevOpsPool \
            --replace

sudo ./svc.sh install || true
sudo ./svc.sh stop || true
sudo ./svc.sh start

The problem is that when I have it set up in my node pool, it is not executing. I am doing that through this Terraform configuration:

resource "azurerm_virtual_machine_scale_set_extension" "pipeliens_vms_custom_script" {
  name                         = "TestingExtension"
  virtual_machine_scale_set_id = azurerm_linux_virtual_machine_scale_set.pipelines_vms.id
  publisher                    = "Microsoft.Azure.Extensions"
  type                         = "CustomScript"
  type_handler_version         = "2.0"

  settings = jsonencode({
    "commandToExecute" = file(abspath("./scripts/pipelinesVmStartupScript.sh"))
  })
}

I just need to figure out how to view the failures from the script or figure out why it's not launching the way I expect it to.

Update 3

Switched to this an it runs:

resource "azurerm_linux_virtual_machine_scale_set" "pipelines_vms" {
  ...
  custom_data                     = base64encode(file(abspath("./scripts/pipelinesVmStartupScript.sh")))

Just need to figure out why the config.sh run is failing.

Update 4

Script fails with:

Must not run with sudo

I am going to create a user in the script and have it run as that user. This should do the trick.

The custom_data part works peachy keen. This is the script I needed to get it running:

#!/bin/sh
set -e

sudo apt-get update
sudo apt install -y jq apt-transport-https ca-certificates curl gnupg lsb-release

sudo apt install -y liblttng-ust0 libkrb5-3 zlib1g

sudo apt install -y libssl1.1 || \
  sudo apt install -y libssl1.0.2 || \
  sudo apt install -y libssl1.0.0

sudo apt install -y libicu67 || \
  sudo apt install -y libicu66 || \
  sudo apt install -y libicu63 || \
  sudo apt install -y libicu60 || \
  sudo apt install -y libicu57 || \
  sudo apt install -y libicu55 || \
  sudo apt install -y libicu52

sudo addgroup vsts
sudo useradd -s /bin/bash -d /home/vsts -m -g vsts vsts
sudo usermod -aG sudo vsts

sudo mkdir /vsts-install
cd /vsts-install

sudo curl -sSL https://vstsagentpackage.azureedge.net/agent/2.187.2/vsts-agent-linux-x64-2.187.2.tar.gz --output vsts.tgz
sudo tar -zxf vsts.tgz
sudo chown -R vsts:vsts /vsts-install

sudo su -c "./config.sh --unattended --url https://dev.azure.com/${MY_ORGANIZATION} --auth pat --token ${MY_INJECTED_TOKEN} --pool ${MY_POOL_NAME} --replace 2>&1 | tee /tmp/CONFIG_LOG" vsts

sudo ./svc.sh install || true
sudo ./svc.sh stop || true
sudo ./svc.sh start

cd ..