Alert configuration error

Plugins Alerting
1

I am trying to create an alert for 404 request using POST api for opendistro,
My request looks like this:

GET _opendistro/_alerting/monitors/89O5w2kBky_xhYOzSrqI

POST _opendistro/_alerting/monitors
{
  "monitor" : {
    "type" : "monitor",
    "name" : "test-monitor-404",
    "enabled" : false,
    "enabled_time" : null,
    "schedule" : {
      "period" : {
        "interval" : 1,
        "unit" : "MINUTES"
      }
    },
    "inputs" : [
      {
        "search" : {
          "indices" : [
            "filebeat-*"
          ],
          "query" : {
            "size" : 0,
            "query" : {
              "bool" : {
                "must": [
                  {
                    "range" : {
                      "@timestamp" : {
                        "from" : "{{period_end}}||-5m",
                        "to" : "{{period_end}}",
                        "include_lower" : true,
                        "include_upper" : true,
                        "format" : "epoch_millis",
                        "boost" : 1.0
                      }
                    }
                  },
                  {
                    "match": {
                      "status": "404"
                    }
                  }
                ],
                "adjust_pure_negative" : true,
                "boost" : 1.0
              }
            },
            "aggregations" : { }
          }
        }
      }
    ],
    "triggers" : [
      {
        "id" : "ldO6w2kBky_xhYOzeLz9",
        "name" : "test",
        "severity" : "1",
        "condition" : {
          "script" : {
            "source" : "ctx.results[0].hits.total > 1",
            "lang" : "painless"
          }
        },
        "actions" : [
          {
            "throttle_period_in_millis" : 300000,
            "name" : "test",
            "destination_id" : "7NOtw2kBky_xhYOzYqqB",
            "subject_template" : {
              "source" : "test",
              "lang" : "mustache"
            },
            "message_template" : {
              "source" : "test message ",
              "lang" : "mustache"
            }
          }
        ]
      }
    ]
  }
}

But whenever I try to run it from dev tools, it is throwing an error:

{
  "error": {
    "root_cause": [
      {
        "type": "uninitialized_property_access_exception",
        "reason": "lateinit property name has not been initialized"
      }
    ],
    "type": "uninitialized_property_access_exception",
    "reason": "lateinit property name has not been initialized"
  },
  "status": 500
}

What am I missing?

2

Hi Nishant,

It appears you did a GET request first, and then attempted to POST the same document again. This will not work due to the way that we store monitors on the back-end you need to change your root level field and remove "monitor": { ... }. You also have another field that we do not support under actions, which is "throttle_period_in_millis" : 300000. Once these two are removed the monitor is created just fine. I am curious on how you got the throttle_period_in_millis field. Is that located somewhere in our documentation?

Here is the complete request I think you are looking for:

    "name" : "test-monitor-404",
    "enabled" : false,
    "enabled_time" : null,
    "schedule" : {
      "period" : {
        "interval" : 1,
        "unit" : "MINUTES"
      }
    },
    "inputs" : [
      {
        "search" : {
          "indices" : [
            "filebeat-*"
          ],
          "query" : {
            "size" : 0,
            "query" : {
              "bool" : {
                "must": [
                  {
                    "range" : {
                      "@timestamp" : {
                        "from" : "{{period_end}}||-5m",
                        "to" : "{{period_end}}",
                        "include_lower" : true,
                        "include_upper" : true,
                        "format" : "epoch_millis",
                        "boost" : 1.0
                      }
                    }
                  },
                  {
                    "match": {
                      "status": "404"
                    }
                  }
                ],
                "adjust_pure_negative" : true,
                "boost" : 1.0
              }
            },
            "aggregations" : { }
          }
        }
      }
    ],
    "triggers" : [
      {
        "id" : "ldO6w2kBky_xhYOzeLz9",
        "name" : "test",
        "severity" : "1",
        "condition" : {
          "script" : {
            "source" : "ctx.results[0].hits.total > 1",
            "lang" : "painless"
          }
        },
        "actions" : [
          {
            "name" : "test",
            "destination_id" : "7NOtw2kBky_xhYOzYqqB",
            "subject_template" : {
              "source" : "test",
              "lang" : "mustache"
            },
            "message_template" : {
              "source" : "test message ",
              "lang" : "mustache"
            }
          }
        ]
      }
    ]
}'
3

Hi thanks for the reply and regarding throttle_period_in_millis, it is not mentioned anywhere in documentation, this is one of the attribute used in xpack of ES used to prevent to send alerts repeatedly. What it does is it will send alerts after the throttle period has been completed after previous alert and the condition for monitoring is still satisfying. So do opendistro-es has something similar to that?

4

Nishant,

No opendistro does not have a functionality like this. The way we limit sending multiple notifications is slightly different.

Lets say you have a monitor and the trigger goes into alarm, it will then go ahead and send a notification to the actions you configured, and will continue to due so as often as the monitors trigger is ran. In order to prevent another notification you would acknowledge the alert. At this point no more notifications will be sent, once the issue is resolved and the trigger evaluates to false the alert will move into the completed state. You will be notified the next time the trigger evaluates to true again.

If you would like functionality like you mentioned, please feel free to create an issue on github.

6

Hi lucas whenever i’m using the following post call for creating monitor ,even though i have mentioned monitor to schedule for every 5 mintues(or any other value). when the monitor is created i can see it for every 1 minute in the UI . can please help on this issue?

  • List item POST _opendistro/_alerting/monitors

{

"type" : "monitor" ,

"name" : "CPU_Utilization" ,

"enabled" : true ,

"schedule" : {

"period" : {

"interval" : 5 ,

"unit" : "MINUTES"

}

},

"inputs" : [

{

"search" : {

"indices" : [

"metricbeat-*"

],

"query" : {

"size" : 0 ,

"query" : {

"bool" : {

"must_not" : [{ "match" : {

"host.os.name.keyword" : "Windows Server 2019 Datacenter"

}

}],

"filter" : [

{

"range" : {

"@timestamp" : {

"from" : "now-5m" ,

"to" : "now" ,

"include_lower" : true ,

"include_upper" : true ,

"boost" : 1.0

}

}

}

],

"adjust_pure_negative" : true ,

"boost" : 1.0

}

},

"aggregations" : {

"hosts" : {

"terms" : {

"field" : "agent.hostname.keyword" ,

"size" : 100 ,

"min_doc_count" : 1 ,

"shard_min_doc_count" : 0 ,

"show_term_doc_count_error" : false ,

"order" : [

{

"_count" : "desc"

},

{

"_key" : "asc"

}

]

},

"aggregations" : {

"max_cpu_usage" : {

"max" : {

"field" : "system.cpu.total.norm.pct"

}

},

"percentage" : {

"bucket_script" : {

"buckets_path" : {

"my_var1" : "max_cpu_usage"

},

"script" : {

"source" : "(params.my_var1)*100" ,

"lang" : "painless"

},

"gap_policy" : "skip"

}

},

"sales_bucket_filter" : {

"bucket_selector" : {

"buckets_path" : {

"TotalPercentage" : "percentage"

},

"script" : {

"source" : "params.TotalPercentage >= 80" ,

"lang" : "painless"

},

"gap_policy" : "skip"

}

}

}

}

}

}

}

}

],

"triggers" : [

{

"name" : "CPU_Utilization >=80%" ,

"severity" : "1" ,

"condition" : {

"script" : {

"source" : "" "

if (ctx.results[ 0 ].aggregations.hosts.buckets==[])

{ return false ; } else { return true ;}

"" ",

"lang" : "painless"

}

},

"actions" : [

{

"name" : "CPU_Utilization >=80%" ,

"destination_id" : "xxxxxxxxxxxxxxxxxxxx" ,

"message_template" : {

"source" : "" "

Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.

- Trigger: {{ctx.trigger.name}}

- Severity: {{ctx.trigger.severity}}

- Period start: {{ctx.periodStart}}

- Period end: {{ctx.periodEnd}}

- Hosts:

{{#ctx.results. 0 .aggregations.hosts.buckets}}

* {{key}} - {{percentage.value}}%

{{/ctx.results. 0 .aggregations.hosts.buckets}}

"" ",

"lang" : "mustache"

},

"throttle_enabled" : false ,

"subject_template" : {

"source" : "[OWEO-Staging-KIBANA] CPU Utilization of nodes >=80%" ,

"lang" : "mustache"

}

}

]

}

]

}