azureazure-log-analytics

How to set up a Low Disk Space alert with the new Azure Monitor agent to trigger when a specific disk drive is low on free space?

For some time now, I have used the Log Analytics agents (MMA.OMS) to collect perf logs and send email alerts when a VM's free disk space % reached a low number. I use a custom log search such as this to trigger the alert:

Perf
| where CounterValue <= 10
| where InstanceName == "C:" 
| where CounterName == "% Free Space"

This worked very well and accomplished exactly what I needed. However, I recently started seeing this message in my Log Analytics workspace telling me that this feature will no longer be supported by the end of August next year, and I should migrate to the new "Azure Monitor Windows Agent" feature: Warning about Log Analytics agent being sunsetted

I tried to set up the new Azure Monitor Windows Agent by creating a Data Collection Rule in the Log Analytics workspace->Agents->Data Collection Rules. I created a Data Collection Rule with a Performance Counters Data source, collecting the custom performance counter for "\LogicalDisk(_Total)% Free Space". This almost accomplishes what I need it to do.

Now when I query the perf logs, I only get log entries where InstanceName is "_Total" instead of also having log entries with InstanceName for specific drives like "C:". While this is fine for VMs that only have one disk drive, this is useless for VMs that have multiple disk drives, including VM sizes that include a large temporary disk drive. I had to change the custom log search to this, in order to get any results:

Perf
| where CounterValue <= 10
| where CounterName == "% Free Space"

So, with my new Azure Monitor logs, if a VM has a C: drive that is at 10% free space but it also has a D: drive that is at 99% free space, the log entries show the total free space of both drives added together, showing something like 46% total free space. This does not accomplish my needs of being alerted when a specific drive is running low on space.

Is there any way to set up an alert for low disk space for specific disk drives using the new Azure Monitor/Data Collection Rules?

Solution

  • While the new log agent does not seem to have the ability in the Azure Portal UI to generate per/drive log entries, you can create the granularity you need using the CLI. I found it easiest to follow this procedure:

    1. Utilize the Azure Portal to create a Data Collection Rule to collect disk performance metrics every 60 seconds.

    2. Modify that Data Collection Rule using the following PowerShell Command through the Azure Cloud Shell:

    az monitor data-collection rule performance-counter update --rule-name {DataCollectionRuleName} --resource-group {ResourceGroupName} --name perfCounterDataSource60 --counter-specifiers "\LogicalDisk(*)\% Free Space"

    Note that in the command 'perfCounterDataSource60' is the automatically generated name of the performance counter created in step one. It is possible that the automatically generated names might vary in the future, or if you use a different interval then 60 seconds.

    By doing this, you'll get Perf log entries for the individual drives in addition to the _total

    Note: I got the magic bits for this solution from the following post on Reddit, Op /u/maldoDev answered his own question: https://www.reddit.com/r/AZURE/comments/11g2lec/azure_monitor_agent_cant_monitor_logical_disk/