Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch 3.5/OSD 3.5/Win10/Google Chrome
Describe the issue:
I am using OpenSearch 3.5 and need assistance configuring an Alerting Monitor.
I need to create one monitor with three triggers (A, B, and C) for detecting critical events in Proxmox logs and sending email notifications.
Trigger A — Critical Keywords
The trigger should fire when the message field contains any of the following words, case-insensitive:
error, warning, alert, failed, unavailable, lost, wrong, unable, invalid, failure
The trigger must not fire if the message contains any of these substrings, case-insensitive:
disabling inotify, user config - ignore, _put_session, PVE ticket
Trigger B — Services and Specific Messages
The trigger should fire when the service field equals any of the following:
pvedaemon, pveproxy, corosync, pmxcfs, pve-ha-crm, pve-ha-lrm
AND the message field contains any of the following substrings:
not exist, Stopped corosync, Stopped pve-ha-crm, Stopped pve-ha-lrm, changed from 'online', => lost_manager_lock, => 'unknown', fence, missing resource
Matching should be case-insensitive.
Trigger C — Frequency-Based Detection
The trigger should fire when any one of the following strings occurs more than once within a 24-hour period:
wait_for_quorum, changed from 'online'
Each string must be counted separately. Therefore:
-
wait_for_quorumoccurs twice → trigger -
changed from 'online'occurs twice → trigger -
each occurs once → do not trigger
When any of the three triggers fires, the monitor should send an email notification.
Please provide the complete recommended configuration and Painless trigger conditions for OpenSearch 3.5, including the correct way to implement the 24-hour frequency condition in Trigger C.
Configuration:
Monitor configuration
{
"name": "GRIU.Proxmox service monitoring",
"type": "monitor",
"monitor_type": "query_level_monitor",
"enabled": true,
"schedule": {
"period": {
"unit": "MINUTES",
"interval": 5
}
},
"inputs": [
{
"search": {
"indices": [
"griu-*"
],
"query": {
"size": 100,
"query": {
"range": {
"@timestamp": {
"from": "now-5m",
"to": "now",
"include_lower": true,
"include_upper": true
}
}
},
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
}
}
]
}
Trigger A — Critical Keywords:
Name: Proxmox HA failure
Type: query_level_trigger
Severity: 2
Condition:
def bad = [
"error",
"warning",
"alert",
"failed",
"unavailable",
"lost",
"wrong",
"unable",
"invalid",
"failure"
];
def ignore = [
"disabling inotify",
"user config - ignore",
"_put_session",
"PVE ticket"
];
for (def hit : ctx.results[0].hits.hits) {
String msg = hit._source.message != null
? hit._source.message.toString().toLowerCase()
: "";
boolean found = false;
for (def word : bad) {
if (msg.contains(word)) {
found = true;
break;
}
}
if (!found) {
continue;
}
for (def word : ignore) {
if (msg.contains(word)) {
found = false;
break;
}
}
if (found) {
return true;
}
}
return false;
Trigger B — Proxmox Services and Specific Messages:
Name: Proxmox general errors
Type: query_level_trigger
Severity: 2
Condition:
def bad = [
"error",
"warning",
"alert",
"failed",
"unavailable",
"lost",
"wrong",
"unable",
"invalid",
"failure"
];
def ignore = [
"disabling inotify",
"user config - ignore",
"_put_session",
"PVE ticket"
];
for (def hit : ctx.results[0].hits.hits) {
String msg = hit._source.message == null
? ''
: hit._source.message.toString().toLowerCase();
boolean hasBad = false;
for (def w : bad) {
if (msg.contains(w)) {
hasBad = true;
break;
}
}
if (!hasBad) {
continue;
}
boolean hasIgnore = false;
for (def x : ignore) {
if (msg.contains(x)) {
hasIgnore = true;
break;
}
}
if (!hasIgnore) {
return true;
}
}
return false;
Trigger C — Frequency-Based Detection:
Name: Quorum repeats
Type: query_level_trigger
Severity: 1
Condition:
int quorum = 0;
int online = 0;
for (def hit : ctx.results[0].hits.hits) {
String msg = hit._source.message == null
? ''
: hit._source.message.toString().toLowerCase();
if (msg.contains('wait_for_quorum')) {
quorum++;
}
if (msg.contains("changed from 'online'")) {
online++;
}
}
return quorum > 1 || online > 1;