azureazure-devopsazure-alertsazure-agent

Azure Alerts: Setting up an alert for Scale Sets

I would like to set up an alert for the following scenario:

A pipeline agent that has been scaled up by a scale set runs for more than 1 hour.

Unfortunately, I only see the possibility to set alarms for the entire VMSS but not for its individual agents.

Thank you

Solution

  • To set alarms for an individual VM in VMSS, you can refer to the following steps.

    1. Go to Monitor -> Alerts -> Create -> Alert rule. enter image description here

    2. Click the drop-down button of your target subscription, resource group, and VMSS in sequence. You can set the scope to an individual VM. enter image description here

      Then you can see the scope like this: enter image description here

    However, I haven't seen a Condition that can be used to monitor the running time of the VM.

    Update

    As a workaround, you can create a new pipeline and set the pipeline running on your VMSS as the pipeline resource. When the resource pipeline is completed, the new pipeline will be triggered. In the new pipeline, run REST API Builds - Get to get the total running time of the resource pipeline. If the running time exceeds 60 minutes, then use the task Send Email to send the email to your target account.

    trigger:
- none

resources:
  pipelines:
  - pipeline: TriggerBuild # Name of the pipeline resource.
    source: CIBuild # The name of the pipeline referenced by this pipeline resource.
    project: {The project name of your resource pipeline} # Required only if the source pipeline is in another project
    trigger: true 

pool:
  vmImage: windows-latest

steps:
- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      $header = @{'Authorization' = 'Bearer ' + "$(System.AccessToken)"}
      $url = "$(System.CollectionUri)$(System.TeamProjectId)/_apis/build/builds/$(resources.pipeline.TriggerBuild.runID)?api-version=7.1-preview.7"
      $result = Invoke-RestMethod -Uri $url -Headers $header -ContentType "application/json" -Method Get

      # calculate the total time of the build
      $time = [datetime]$result.finishTime - [datetime]$result.startTime
      $buildtime= $time.TotalMinutes
      
      if($buildtime -gt 60 ){ echo "##vso[task.setvariable variable=sendnotification]True" }

- task: SendEmail@1
  condition: eq(variables.sendnotification, 'True')
  inputs:
  ...

    In the PowerShell@2 task, if the running time of the resource pipeline exceeds 60 minutes, create an output variable sendnotification and set its value to true. Use condition to limit the SendEmail@1 task to run only when the value of sendnotification is "True".