Unable To Enable Prometheus Lifecycle API In Azure Container Instances

I've been having difficulty adding the --web.enable-lifecycle flag to prometheus, to enable the lifecycle api.

I have this working locally with docker compose, like so:

  prometheus:
    image: prom/prometheus
    volumes:
      - ./tmp/prometheus:/prometheus
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus/targets:/etc/prometheus/targets
    networks:
      - localprom
    ports:
      - 9090:9090
    command:
      - --web.enable-lifecycle
      - --config.file=/etc/prometheus/prometheus.yml

But Azure's YAML format for creating/updating container instances is a little different than docker. Here is my YAML file:

apiVersion: 2021-10-01
location: useast
name: my-prometheus-container-instances
tags: { environment: development }
type: Microsoft.ContainerInstance/containerGroups

properties:
  containers:
    - name: prometheus
      properties:
        image: prom/prometheus:v2.45.6
        command:
          - /bin/prometheus
          - --config.file=/etc/prometheus/prometheus.yml
          - --web.enable-lifecycle
        resources:
          requests:
            cpu: 1
            memoryInGb: 1.5
        ports:
          - port: 9090
        volumeMounts:
          - name: prometheus-fileshare-volume
            mountPath: /etc/prometheus/

  subnetIds:
    - id: MY_SUBNET_ID

  osType: Linux
  volumes:
    - name: prometheus-fileshare-volume
      azureFile:
        shareName: prometheus-fileshare
        readOnly: true
        storageAccountName: prometheus-storage-account
        storageAccountKey: MY_ACCESS_KEY

  RestartPolicy: OnFailure
  ipAddress:
    type: Private
    ports:
      - protocol: tcp
        port: 9090

Note that the Azure YAML file requires - /bin/prometheus in the command array, while docker does not. Without this, prometheus will not start after deploying. I've also confirmed that the --config.file command is working, wondering if there is something else that may be stopping me from enabling the lifecycle API after deploying to Azure.

To enable the Prometheus lifecycle API in an Azure Container Instance using a YAML configuration, you need to make sure that the command array is properly configured and that all necessary parameters are correctly passed.

Create a storage account

az storage account create --name prometheusstorageacct --resource-group arkorg --location eastus --sku Standard_LRS

Create a fileshare

az storage share create --name prometheusfileshare --account-name prometheusstorageacct

Get the storage key

Create your Prometheus Configuration Files. I am just using a basic example, you edit as per your own requirement.

prometheus.yml:

global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds.

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Example job to scrape metrics from a target
  - job_name: 'example'
    static_configs:
      - targets: ['example.com:80']

targets.yml:

- targets:
  - 'target1.example.com:9100'
  - 'target2.example.com:9100'

Upload Configuration Files to Azure File Share

az storage file upload --share-name prometheusfileshare --source ./prometheus.yml --account-name prometheusstorageacct --account-key $STORAGE_KEY
az storage file upload --share-name prometheusfileshare --source ./targets.yml --account-name prometheusstorageacct --account-key $STORAGE_KEY

Create the Azure Container Instance

apiVersion: 2021-10-01
location: eastus
name: my-prometheus-container-instances
type: Microsoft.ContainerInstance/containerGroups
properties:
  containers:
    - name: prometheus
      properties:
        image: prom/prometheus:v2.45.6
        command:
          - /bin/prometheus
          - --config.file=/etc/prometheus/prometheus.yml
          - --web.enable-lifecycle
        resources:
          requests:
            cpu: 1
            memoryInGb: 1.5
        ports:
          - port: 9090
        volumeMounts:
          - name: prometheus-fileshare-volume
            mountPath: /etc/prometheus/
  osType: Linux
  volumes:
    - name: prometheus-fileshare-volume
      azureFile:
        shareName: prometheusfileshare
        readOnly: true
        storageAccountName: prometheusstorageacct
        storageAccountKey: ...6tNg==
  restartPolicy: OnFailure
  ipAddress:
    type: Public
    ports:
      - protocol: tcp
        port: 9090

Deploy the Azure Container Instance

az container create --resource-group arkorg --file prometheus-aci.yaml

Verify and access Prometheus

az container show --resource-group arkorg --name my-prometheus-container-instances

access prometheus by getting the IP

az container show --resource-group arkorg --name my-prometheus-container-instances --query "ipAddress.ip" --output tsv

go to your browser and put the IP you see as output for the above command

http://<container-ip>:9090.

you can verify your lifecycle using something like

curl -X POST http://52.188.24.225:9090/-/reload

Reference-

prometheus | Prometheus