Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch 2.9.0
Describe the issue:
I have created opensearch alert for monitoring URLs using heartbeat service. I have created 2 triggers in the same alert. one for urls when its down and another one when its up. So when one trigger is in alert state, other one will be in recovery state. Here the issue is that one trigger mostly url up trigger triggering inconsitently. Also attaching the screenshot of the graph indicating the inconsistency of url up trigger.
Configuration:
Below is the extraction query
/////////////////////////////////////
{
“size”: 0,
“query”: {
“bool”: {
“must”: [
{
“terms”: {
“monitor.status.keyword”: [
“down”,
“up”
],
“boost”: 1
}
},
{
“range”: {
“@timestamp”: {
“from”: “now-2m”,
“to”: “now”,
“include_lower”: true,
“include_upper”: true,
“boost”: 1
}
}
},
{
“bool”: {
“should”: [
{
“wildcard”: {
“url.domain.keyword”: {
“wildcard”: “example.com”,
“boost”: 1
}
}
}
],
“adjust_pure_negative”: true,
“boost”: 1
}
}
],
“adjust_pure_negative”: true,
“boost”: 1
}
},
“aggregations”: {
“up_services”: {
“filter”: {
“term”: {
“monitor.status.keyword”: {
“value”: “up”,
“boost”: 1
}
}
},
“aggregations”: {
“domains”: {
“terms”: {
“field”: “url.domain.keyword”,
“size”: 10,
“min_doc_count”: 1,
“shard_min_doc_count”: 0,
“show_term_doc_count_error”: false,
“order”: [
{
“_count”: “desc”
},
{
“_key”: “asc”
}
]
}
}
}
},
“down_services”: {
“filter”: {
“term”: {
“monitor.status.keyword”: {
“value”: “down”,
“boost”: 1
}
}
},
“aggregations”: {
“domains”: {
“terms”: {
“field”: “url.domain.keyword”,
“size”: 10,
“min_doc_count”: 1,
“shard_min_doc_count”: 0,
“show_term_doc_count_error”: false,
“order”: [
{
“_count”: “desc”
},
{
“_key”: “asc”
}
]
}
}
}
}
}
}
//////////////////////////////////////////
Below is the trigger condition
URL UP Trigger
//////////////////
if ( ctx.results[0].aggregations.down_services.doc_count < 1 && ctx.results[0].aggregations.up_services.doc_count > 0) {
for (bucket in ctx.results[0].aggregations.down_services.domains.buckets) {
if (bucket.doc_count > 0) {
return false;
}
}
for (bucket in ctx.results[0].aggregations.up_services.domains.buckets) {
if (bucket.doc_count > 0) {
return true;
}
}
}
return false;
////////////////////
URL Down Trigger
///////////////////
if ( ctx.results[0].aggregations.down_services.doc_count > 0 && ctx.results[0].aggregations.up_services.doc_count < 1) {
for (bucket in ctx.results[0].aggregations.down_services.domains.buckets) {
if (bucket.doc_count > 0) {
return true;
}
}
}
return false;
Relevant Logs or Screenshots:
