In AKS we dynamically create and use persistent volume with azure disk. We setup initial size 20G for s service and we ran out of space. Is there any way that we can setup a disk alert for dynamic disks to be informed?
Starting with agent version ciprod10052020, Container insights integrated agent now supports monitoring PV (persistent volume) usage : Reference
Create metric alerts :
To alert on system resource issues when they are experiencing peak demand and running near capacity, with Container insights you would create a log alert based on performance data stored in Azure Monitor Logs. For refernce see alerts-metric-overview .
Container insights includes Average disc usage % and Average persistent volume usage% alerts from among many alerts.
Log-based alerts:
Log alerts run queries on Log Analytics data : Log alerts and prerquisites
Create an alert rule for container resource utilization - refer : create an alert rule
Note: To create an alert rule for container resource utilization requires you to switch to a new log alerts API as described in switch API preference for log alerts
To Query to alert when free disk space on cluster nodes exceeds a threshold: see last section of resource-utilization-log-search-queries.