Alert is Not triggering

Learner · March 18, 2019, 10:42am

I have created monitor and trigger. When i am triggering manually , it is getting trigger. But automatic not triggering. “schedule_status” : showing red

{
“size”: 1,
“query”: {
“match”: {
“event_id”: {
“query”: “4720”,
“operator”: “OR”,
“prefix_length”: 0,
“max_expansions”: 50,
“fuzzy_transpositions”: true,
“lenient”: false,
“zero_terms_query”: “NONE”,
“auto_generate_synonyms_phrase_query”: true,
“boost”: 1
}
}
}
}

“cluster_name” : “elasticsearch”,
“opendistro.scheduled_jobs.enabled” : true,
“scheduled_job_index_exists” : true,
“scheduled_job_index_status” : “green”,
“nodes_on_schedule” : 0,
“nodes_not_on_schedule” : 1,
“nodes” : {
“4jPUFB8aQuuTWcgpHyhOxw” : {
“name” : “4jPUFB8”,
“schedule_status” : “red”,
“roles” : [
“MASTER”,
“DATA”,
“INGEST”

jinsoor-amzn · March 18, 2019, 5:33pm

Hi,

Can you please update with step by step on how to reproduce the issue? That is the best way for us to dive deep into the issue.

If it can’t be reproduced easily please post the answers to following questions.

All the information about the Monitor.
- curl localhost:9200/_opendistro/_alerting/monitors/<MonitorJobId>
Response of alerts with regards to the Monitor.
- curl localhost:9200/.opendistro-alerting-alerts/_search?pretty -H 'Content-Type: application/json' -d '{"query": {"terms": {"_routing": ["<MonitorJobId>"]}}}'
Opendistro Thread pool details
- curl localhost:9200/_cat/thread_pool/opendistro_monitor_runner?v\&h=id,name,active,rejected,completed
Full information about the Alerting stats
- curl localhost:9200/_opendistro/_alerting/stats
Any errors that is shown in the Elasticsearch log.

Learner · March 19, 2019, 7:36am

Hi it seems alerts is just triggering one time and after that alerts is not triggering for any new events.
Mean , i acknowledge one alerts , then i created same event again for the same alerts , but alerts didn’t trigger.

1.)

"_id" : "BmO3j2kBBSkdQvmwHoOt",
  "_version" : 6,
  "monitor" : {
    "type" : "monitor",
    "name" : "User Created",
    "enabled" : true,
    "enabled_time" : 1552894335947,
    "schedule" : {
      "period" : {
        "interval" : 1,
        "unit" : "MINUTES"
      }
    },
    "inputs" : [
      {
        "search" : {
          "indices" : [
            "winlogbeat-2019.03.19"
          ],
          "query" : {
            "size" : 100,
            "query" : {
              "match" : {
                "event_id" : {
                  "query" : "4720",
                  "operator" : "OR",
                  "prefix_length" : 0,
                  "max_expansions" : 50,
                  "fuzzy_transpositions" : true,
                  "lenient" : false,
                  "zero_terms_query" : "NONE",
                  "auto_generate_synonyms_phrase_query" : true,
                  "boost" : 1.0
                }
              }
            }
          }
        }
      }
    ],
    "triggers" : [
      {
        "id" : "C2O3j2kBBSkdQvmwjINP",
        "name" : "User Created",
        "severity" : "2",
        "condition" : {
          "script" : {
            "source" : "ctx.results[0].hits.total > 0",
            "lang" : "painless"
          }
        },
        "actions" : [ ]
      }
    ],
    "last_update_time" : 1552979590597
  }
}

2.)
{
  "took" : 18,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : ".opendistro-alerting-alerts",
        "_type" : "_doc",
        "_id" : "EWO4j2kBBSkdQvmwBoNM",
        "_score" : 1.0,
        "_routing" : "BmO3j2kBBSkdQvmwHoOt",
        "_source" : {
          "monitor_id" : "BmO3j2kBBSkdQvmwHoOt",
          "monitor_version" : 2,
          "monitor_name" : "User Created",
          "trigger_id" : "C2O3j2kBBSkdQvmwjINP",
          "trigger_name" : "User Created",
          "state" : "ACKNOWLEDGED",
          "error_message" : null,
          "alert_history" : [ ],
          "severity" : "2",
          "start_time" : 1552894395778,
          "last_notification_time" : 1552894456006,
          "end_time" : null,
          "acknowledged_time" : 1552894473452
        }
      }
    ]
  }
}

3.) 
id                     name                      active rejected completed
4jPUFB8aQuuTWcgpHyhOxw opendistro_monitor_runner      0        0        66

4.){
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "elasticsearch",
  "opendistro.scheduled_jobs.enabled" : true,
  "scheduled_job_index_exists" : true,
  "scheduled_job_index_status" : "green",
  "nodes_on_schedule" : 1,
  "nodes_not_on_schedule" : 0,
  "nodes" : {
    "4jPUFB8aQuuTWcgpHyhOxw" : {
      "name" : "4jPUFB8",
      "schedule_status" : "green",
      "roles" : [
        "MASTER",
        "DATA",
        "INGEST"
      ],
      "job_scheduling_metrics" : {
        "last_full_sweep_time_millis" : 189817,
        "full_sweep_on_time" : true
      },
      "jobs_info" : {
        "BmO3j2kBBSkdQvmwHoOt" : {
          "last_execution_time" : 1552980675947,
          "running_on_time" : true
        },
        "BS_OlGkB8nQEAav636lN" : {
          "last_execution_time" : 1552980679375,
          "running_on_time" : true
        }
      }
    }
  }
}

Yogesh · March 19, 2019, 4:17pm

Did you test monitor without any action creation?
Is it triggering after first trigger.

jinsoor-amzn · March 19, 2019, 5:14pm

Hi Learner,

From your example you have a monitor running every 1 minute and a trigger that alerts whenever it finds at least one document matching the query set up in your monitor. You can only ever have one active alert for a monitor-trigger pair, it will just update the current active alert if it already exists. When you acknowledge an alert it still remains in an “active” state, acknowledging is a way to reduce noise so that you aren’t notified constantly. When the alert is acknowledged, any “action” on the Trigger will be skipped.
Once the existing “acknowledged” alert is completed, the monitor will be back to “normal” state and can create new active alert when trigger condition is met.

Learner · March 20, 2019, 6:31am

How to complete alerts. Once i acknowledge alerts , i am not able to select that alerts again.

jinsoor-amzn · March 20, 2019, 6:28pm

The alert will be in complete state when the “corresponding trigger evaluates to false.” You can also view the details about different Alerting State in our Opendistro Alerting Document.

Yogesh · March 20, 2019, 9:30pm

Hi, Jinsoor

I am also looking for this.how I can evaluate trigger to false positive so that alerts will be completed.i gone through documents but I didn’t get much stuff about it.
From which tab I can perform trigger evaluation, from dashboard or monitor. In dashboard , once I acknowledge alerts I am not able to select those alerts again to make it complete.

dbbaughe · March 22, 2019, 5:08am

Hi Yogesh,

We do not have the notion of a “false alert” in alerting. Acknowledging an alert means that you have acknowledged there is an active alert, are taking steps to fix it and do not need to be notified again (if you set up any actions for it). Fixing it can be by fixing the underlying reason that caused it to trigger or modifying the trigger itself so on the next execution it evaluates to is false. We do not support manually completing an alert, since you would just have a brand new alert and be notified the next time the monitor runs if it still evaluates to true.

That being said, the notion of a “false alert” and how we could handle such scenarios is definitely interesting, will look into it more

Learner · March 22, 2019, 5:53am

Thanks , as i understood , as of now alerts function will not work like SIEM alerts or real time alerts. My expectation is alerts should show on dashboard or notify on each and every condition match irrespective of whether it is active or acknowledge or completed. If condition matching two time then two alerts should be on dashboard , not only one.

carlmead · March 22, 2019, 3:11pm

Hi Learner - The Alerting project is providing real-time alerts it is just not re-notifying on every occurrence of an Active alert once you have acknowledged it - which is the equivalent of saying you are working on it. Our thinking was not to spam folks (probably because we hate it when we get spammed) but I can certainly understand the counter-point. Looks like someone created an issue for the same on GitHub: Alerts not reflecting on dashboard · Issue #10 · opendistro-for-elasticsearch/alerting · GitHub. Feel free to plus one it. It may be a good candidate to make this an option as I can see how folks could like it both ways.

Yogesh · March 22, 2019, 4:17pm

How I can complete alerts , so that alert will get trigger on next match.
Once alert acknowledged , I am not able to select that alerts to complete and so alert is not getting trigger or notify on second match.
Is there any way. I can complete alert. I gone through documents but no luck.

dbbaughe · March 22, 2019, 4:56pm

Hi Yogesh, if you would like to be notified every time the alert is triggered then you can choose to not acknowledge the alert and you will be notified every time. Acknowledging is simply a way to stop notifications for that ongoing alert, which seems to be the opposite of what you want to do.

Yogesh · March 22, 2019, 6:18pm

My simple requirement
Alert Name :virus detected , I have created simple query to match virus detected events.
Now first events virus detected from system A after some time virus detected from system b , so expecting two separate alerts on dashboard.

Another example: user created alerts
If two different user is getting created then expecting two separate alerts on dashboard.

How I can achieve this.

.

dbbaughe · March 22, 2019, 6:37pm

Hi Yogesh,

An example of how you could work around your two requirements:

Create a monitor that runs every 5 minute
Write a query using the range filter with the provided period_start period_end parameters we provide to only look at documents from the last 5 minutes and matching your document types.
Use a terms aggregation to create unique buckets per user_id or per system (for virus use case).
Write a trigger that will notify you whenever the aggregation buckets list is > 0
And in the action message, you can notify yourself with how many users were created in the last 5 minutes or how many systems have a virus in the last 5 minutes along with their user_ids/system name respectively.

You are correct in that we do not support an alert per document like you’re looking for, but there are ways to work around it like above. Let me know if I missed anything, thanks

Yogesh · March 27, 2019, 11:30am

Hi Thanks,

I have managed to configured it. And now i am near to my expectation.

Below is my query. Now alerts are getting completed automatically at 5 minute and new alerts are triggering after 5mn on next event match. Now my concern is , two events matched (two different user/system) within 5 mn but still alert is one only. I want alerts on each match irrespective of time span . How i can do it.

{
    "size": 2,
    "query": {
        "bool": {
            "must": [
                {
                    "match": {
                        "event_id": {
                            "query": "4726",
                            "operator": "OR",
                            "prefix_length": 0,
                            "max_expansions": 50,
                            "fuzzy_transpositions": true,
                            "lenient": false,
                            "zero_terms_query": "NONE",
                            "auto_generate_synonyms_phrase_query": true,
                            "boost": 1
                        }
                    }
                }
            ],
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "now-5m",
                            "to": null,
                            "include_lower": true,
                            "include_upper": true,
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {
        "Group_By_TargetUserName": {
            "terms": {
                "field": "event_data.TargetUserName.keyword",
                "size": 10,
                "min_doc_count": 1,
                "shard_min_doc_count": 0,
                "show_term_doc_count_error": false,
                "order": [
                    {
                        "_count": "desc"
                    },
                    {
                        "_key": "asc"
                    }
                ]
            },
            "aggregations": {
                "get_latest": {
                    "terms": {
                        "field": "@timestamp",
                        "size": 1,
                        "min_doc_count": 1,
                        "shard_min_doc_count": 0,
                        "show_term_doc_count_error": false,
                        "order": {
                            "_key": "desc"
                        }
                    }
                }
            }
        }
    }
}

dbbaughe · March 28, 2019, 4:57pm

Hi Yogesh,

We do not currently support alerting on a per document/event level. The example I gave is what you can do to work around that for now (it supports as low as minute intervals).

You can also make a feature request in the issues section of the GitHub repository:

rmy · April 30, 2019, 8:13am

I think the problem is that the retriggering of alerts is missing; as outlined in this feature-request:

https://github.com/opendistro-for-elasticsearch/alerting/issues/42

We have the same issue; we use aggregations to separate the entities, but since there are no retriggering of alarms we are left in the dark. You dont need to be able to create a fully result-aware retrigger-mechanism, a simple “retrigger” with possibly a delay would go a long way of solving the problem.

Selfmade_RuLeZ · May 17, 2019, 8:54am

Hi, I think you have to add a time range inside a bool query. For example:

          "bool": {
            "must": {
                "match" : {
                                your query here
                           }
                "range": {
                    "@timestamp": {
                        "from": "now-1m",
                        "to": null,
                        "include_lower": true,
                        "include_upper": true,
                        "boost": 1
                    }
                  }
                }
              }

This will query your match within only the last one minute. This is how I solved this problem. I hope I can help you

panpanpandas · June 23, 2019, 9:56pm

+1 It would be nice to add retriggering of alerts. We have the same issue for daily triggered events. We use these alerts to let sales follow up with orders, the alerts can be triggered everyday for different orders.

Topic		Replies	Views
2 configured triggers in Alert, but only 1 trigger is getting triggered Alerting	1	611	October 26, 2021
Alerts are not triggered Alerting	9	1315	November 20, 2019
Alerting not work Alerting	9	2018	October 17, 2020
Alert is only sent when I manually "_execute" the monitor Alerting	4	948	December 2, 2020
Open Distro Alerting Plugin Open Source Elasticsearch and Kibana	1	439	February 7, 2023

Alert is Not triggering

Related topics