Alerts are not triggered

Plugins Alerting
1

Hello, I’m setting up alerts, I’ve created monitor and trigger, even very simple one, just to test.

{
  "_id" : "KbbP920BwY-0PRalu0yh",
  "_version" : 2,
  "monitor" : {
    "type" : "monitor",
    "name" : "test",
    "enabled" : true,
    "enabled_time" : 1571820649372,
    "schedule" : {
      "period" : {
        "interval" : 15,
        "unit" : "MINUTES"
      }
    },
    "inputs" : [
      {
        "search" : {
          "indices" : [
            "app-general-*"
          ],
          "query" : {
            "size" : 0,
            "query" : {
              "bool" : {
                "filter" : [
                  {
                    "range" : {
                      "@timestamp" : {
                        "from" : "{{period_end}}||-1h",
                        "to" : "{{period_end}}",
                        "include_lower" : true,
                        "include_upper" : true,
                        "format" : "epoch_millis",
                        "boost" : 1.0
                      }
                    }
                  }
                ],
                "adjust_pure_negative" : true,
                "boost" : 1.0
              }
            },
            "aggregations" : { }
          }
        }
      }
    ],
    "triggers" : [
      {
        "id" : "3LfR920BwY-0PRalKyps",
        "name" : "test-alert",
        "severity" : "1",
        "condition" : {
          "script" : {
            "source" : "ctx.results[0].hits.total > 10",
            "lang" : "painless"
          }
        },
        "actions" : [
          {
            "name" : "test trigger",
            "destination_id" : "7OCvlGsBwY-0PRal181J",
            "subject_template" : {
              "source" : "",
              "lang" : "mustache"
            },
            "message_template" : {
              "source" : "{\"text\": \"AHTUNG!  test \n- Period start: {{ctx.periodStart}}\n- Period end: {{ctx.periodEnd}}\n\", \"chat_id\":\"-1001464722935\"}",
              "lang" : "mustache"
            }
          }
        ]
      }
    ],
    "last_update_time" : 1571820743532
  }
}

So there more then 10k documents add every minute to main index, but no alert triggered

Please advise me, where is the problem

2

Hi @Vinnyard,
Could you share any log output? On Kibana there is also a button to test your script and check the output, it could be that there is a compile / runtime error.

3

Could you please tell me where to get that logs, in trigger i see this

Because i used visual graph, there is no button to check conditions in trigger. But for other alerts i tried - thet trigger - returned true, so it shoudl create alert, but i see no alerts in system (

4

@Vinnyard,
The logs are located in elasticsearch.log on the machine that is running elasticsearch. Depending on if you are using a hosted system you might not have access to them.

5

we have cluster with 4 data nodes and 3 masters and http node with kibana, any suggestions what exactly i should search on logs? Something about alerting?

6

So we have a lot of logs, and i just do not see anything about alerting …

7

What is the output of this query when ran alone against the cluster? I tested this locally and it appears to work fine, but I also have just sample data.
Please also provide the Elasticsearch version you are using.

Thanks!

8

My elk version is 6.7.1
The result is

{
    "_shards": {
        "total": 710,
        "failed": 0,
        "successful": 710,
        "skipped": 705
    },
    "hits": {
        "hits": [],
        "total": 95366,
        "max_score": 0
    },
    "took": 40,
    "timed_out": false
}
9

Hi @Vinnyard,

I once again tested this (previously I tested using 7.1.1 this time using 6.7.1), and everything is working fine.
Here are some follow up questions:

  • Are you using the alerting plugin along with another plugin?
  • Can you share any the log output (elasticsearch.log) from the time the monitor is created to the time it executes ± 1 minute?
    You should see something like this for creation:
[2019-11-12T10:30:56,334][INFO ][c.a.o.a.c.s.JobScheduler ] [r3v6AZN] Scheduling jobId : bvfiYG4Bw2Ei-yhJCIpA, name: Test monitor
[2019-11-12T10:31:21,717][INFO ][c.a.o.a.c.s.JobScheduler ] [r3v6AZN] Descheduling jobId : bvfiYG4Bw2Ei-yhJCIpA
[2019-11-12T10:31:21,718][INFO ][c.a.o.a.c.s.JobScheduler ] [r3v6AZN] Scheduling jobId : bvfiYG4Bw2Ei-yhJCIpA, name: Test monitor

And if nothing goes wrong then @ 2019-11-12T10:31:21,718 + interval (1 minute in my case), there should be no errors.

Thanks,
Lucas Winkelmann

10

We use this plugins

root@ipl:~# curl localhost:9200/_cat/plugins
ipl.elk.h1-ipl.elk.h1 opendistro_alerting 0.9.0.0
ipl.elk.h1-ipl.elk.h1 opendistro_sql 0.9.0.0

Also i looked for anything about JobScheduler
on all servers (http, 3x master, 4x data nodes)
but i found nothing( before that i enebled and disabled monitor):

root@ipl:/var/log/elasticsearch/ipl.elk.h1-ipl.elk.h1# grep JobScheduler ipl-elk.log
root@ipl:/var/log/elasticsearch/ipl.elk.h1-ipl.elk.h1#