Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch: 2.9.0
Describe the issue:
I try to using bucket monitor with smtp .
At the beginning, it works and sent some alerting mail.
After I configured like: action config or schedule in the same monitor few times,
the monitor won’t be triggered any more!
The other bucket monitors(exist or new) have same issue.
There is no error in opensearch log.
Clicking “send test message” below the message text box is work.
Per query monitor still works.
I copy the same monitor and trigger settings to another 2.9.0 opensearch cluster.
It can work fine !
How to fix the issue or where can i check monitor execution history ?
Configuration:
Extration query for metricbeat index.
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"from": "now-10m/m",
"to": "now/m",
"include_lower": true,
"include_upper": true,
"boost": 1
}
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"aggregations": {
"hosts": {
"terms": {
"field": "host.name",
"size": 10,
"min_doc_count": 1,
"shard_min_doc_count": 0,
"show_term_doc_count_error": false,
"order": [
{
"avg_cpu_total_pct": "desc"
},
{
"_key": "asc"
}
]
},
"aggregations": {
"avg_cpu_total_pct": {
"avg": {
"field": "system.cpu.total.norm.pct"
}
},
"cpu_filter": {
"bucket_selector": {
"buckets_path": {
"avgCpu": "avg_cpu_total_pct.value"
},
"script": {
"source": "params.avgCpu >= 0.05",
"lang": "painless"
},
"gap_policy": "skip"
}
}
}
}
}
}
Trigger condition
{
"buckets_path": {
"cpuAvg": "avg_cpu_total_pct"
},
"parent_bucket_path": "hosts",
"script": {
"source": "params.cpuAvg >= 0.1 && params.cpuAvg <= 0.8",
"lang": "painless"
},
"gap_policy": "skip"
}
Message
Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.
- Trigger: {{ctx.trigger.name}}
- Severity: {{ctx.trigger.severity}}
- Period start: {{ctx.periodStart}} (UTC)
- Period end: {{ctx.periodEnd}} (UTC)
- Deduped Alerts:
{{#ctx.dedupedAlerts}}
* Host : {{bucket_keys}} Alert Id : {{id}}
{{/ctx.dedupedAlerts}}
- New Alerts:
{{#ctx.newAlerts}}
* Host : {{bucket_keys}} Alert Id : {{id}}
{{/ctx.newAlerts}}
- Completed Alerts:
{{#ctx.completedAlerts}}
* Host : {{bucket_keys}} Alert Id : {{id}}}
{{/ctx.completedAlerts}}
Relevant Logs or Screenshots:
Only 1 trigger for testing
It can preview some buckets that can be triggered.
