Opensearch Anomaly Detector Custom Expressions

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
v2.11.1

Describe the issue:
I would like to write a Custom expression in the “Anomaly Detection” section of Opensearch Dashboards, but I can’t seem to get m Detector right.

At the moment, I have:

{
    "http-reponse-code": {
        "value_count": {
            "field": "response"
        }
    }
}

It will count all the messages in the index, that have a http-response-code.

Now, I would like to limit those to only count responses, that are in a certain range (“400” to “499”). I can’t seem to get that right.

Thought about something like this:

{
  "query": { 
    "value_count": { 
      "filter": [ 
        { "term":  { "field": "response"}},
        { "range": { "response": { "gte": 400, "lte": 499 }}}
      ]
    }
  }
}

When I click “Preview”, Opensearch Dashboards tells me:

 query error: [1:1209] [value_count] unknown field [filter]

What am I getting wrong?

Is there any documentation for the expressions, I can use in Anomaly Detection?

Any advice highly appreciated!

Configuration:

Relevant Logs or Screenshots:

2

@automator As far as I know, value_count expects a field.
i.e.

{
    "query": {
        "value_count": {
            "field": "<filed_name>"
        }
    }
}
3

Thanks for commenting! I am aware that it accepts a field by default. I was hoping that there is some possibility to do some filtering inside the expression, but it seems like it is not possible, which really is annoying…

4

@automator I’ve tested with just a filter in that custom expression but it complained about an incorrect input field.

5

Hi Pablo, thanks alot for your time! I guess, it is impossible to achieve this in Opensearch Dashboards Anomaly Detection. It seems to only accept a single field for each feature and no “true complex” queries/filters. At least with the current version of Opensearch Dashboards.

6

Can you try sth like

{
    "filtered_output": {
        "filter": {
            "bool": {
                "must": [
                    {
                        "range": {
                            "response": {
                                "gte": 400,
                                "lte": 499
                            }
                        }
                    }
                ],
                "adjust_pure_negative": true,
                "boost": 1
            }
        },
        "aggregations": {
            "http-response-code": {
                "value_count": {
                    "field": "response"
                }
            }
        }
    }
}

?

2 Likes
7

Wow! Thank you so much, kaituo! This actually seems to work for me! :smiley:

8

nice