anomaly-detectionopensearch-dashboards

Opensearch Anomaly Detector Custom Expressions


I would like to write a Custom expression in Opensearch Dashboards, but I can't seem to get it right.

At the moment, I have:

{
    "http-reponse-code": {
        "value_count": {
            "field": "response"
        }
    }
}

It will count all the messages in the index, that have a http-response-code.

Now, I would like to limit those to only count responses, that are in a certain range ("400" to "499"). I can't seem to get that right.

Thought about something like this:

{
  "query": { 
    "value_count": { 
      "filter": [ 
        { "term":  { "field": "response"}},
        { "range": { "response": { "gte": 400, "lte": 499 }}}
      ]
    }
  }
}

When I click "Preview", Opensearch Dashboards tells me:

 query error: [1:1209] [value_count] unknown field [filter]

What am I getting wrong?

Is there any documentation for the expressions, I can use in Anomaly Detection?

Any advice highly appreciated!


Solution

  • filter_query and feature_attributes are entirely two different stanzas for building-up a anomaly detector.

    While using the UI you setup the Data Filter in the Detector settings section, and the Feature(s) in the Model configuration.

    Or while talking to the API you could have this for the query:

      "filter_query" : {
        "bool" : {
          "filter" : [
            {
              "range" : {
                "response" : {
                  "from" : 400,
                  "to" : 500,
                  "include_lower" : true,
                  "include_upper" : false
                }
              }
            }
          ]
        }
      }
    

    and this for the feature:

      "feature_attributes": [
        {
          "feature_name": "count-status",
          "feature_enabled": true,
          "aggregation_query": {
            "aggs0": {
              "value_count": {
                "field": "response"
              }
            }
          }
        }
      ],
    

    You can get a sample of a full json by listing the existing detectors (possibly created by the UI to begin with)

    $endpoint/_plugins/_anomaly_detection/detectors/_search?pretty
    

    and then show how one looks like using its ID

    $endpoint/_plugins/_anomaly_detection/detectors/$id?pretty
    

    Using the API allows you to make your data filter as you wish (use gte/lt instead of from/to if you like). Using the UI results in somewhat redundant filters (it makes up two filters for a range), but it probably also works.