google-cloud-platformmonitoring

Google Cloud alerting doesn't allow creating an alert for its own metric


I have Firebase API on GCP, but I would like to get alerts when my API returns error for more than 10% of requests.

Something which GCP already has:

enter image description here

It's exactly what I need, first problem is that apparently GCP doesn't understand its own query as it cannot be parsed to builder or promql/mql, which prevents me from importing it to central alert system of grafana.

enter image description here

If I use Google's alerting system, and have this specific alert in different place then all others, apparently it's impossible to create ratio queries in GCP alerting tab.

Also for some reason I cannot use one_of("4xx", "5xx") as it dosen't find responses with "5xx" even tho they are on the chart, but I can use !=200 and it will do same for me.

enter image description here

Is this query useless because I cannot provide alerting for it? What's the point of creating this metric in this case?


Solution

  • First of all I would recommend checking if the log based metric produces any data by itself without the alerting policy.After you check in Monitoring that the log based metric is showing the data that you want you can and then create an alert policy for it.Refer to this for more information.

    If you are having issues with querying specifying status codes it could be due to how the logs are structured or indexed. Also please be noted that GCP’s alerting mechanism has certain limitations especially when dealing with ratios or complex queries. For more information refer to this alerting limits document.

    Since it does not support complex calculations directly you could set up two separate metrics one for total requests and another for error responses.Once this is done you can try creating alerts based on thresholds for these metrics but however it could be done outside GCP or using other monitoring tools.

    And if your GCP’s built in alerting doesn't meet your needs then consider building a custom solution using cloud run or cloud functions.Refer to custom-metrics document for more information on this.