# Help configuring an OpenSearch 3.5 alerting monitor with three triggers

**URL:** <https://forum.opensearch.org/t/help-configuring-an-opensearch-3-5-alerting-monitor-with-three-triggers/28261>\
**Category:** Alerting\
**Tags:** configure, alerting\
**Created:** [August 26, 2026, 7:58am UTC](https://forum.opensearch.org/t/help-configuring-an-opensearch-3-5-alerting-monitor-with-three-triggers/28261 "2026-08-26T07:58:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pan-Vad](https://avatars.discourse-cdn.com/v4/letter/p/6de8d8/32.png) [@Pan-Vad](https://forum.opensearch.org/u/Pan-Vad)\
**Post date:** [August 26, 2026, 7:58am UTC](https://forum.opensearch.org/t/help-configuring-an-opensearch-3-5-alerting-monitor-with-three-triggers/28261/1 "2026-08-26T07:58:14Z")

</div>

**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch 3.5/OSD 3.5/Win10/Google Chrome

**Describe the issue** :

I am using OpenSearch 3.5 and need assistance configuring an Alerting Monitor.

I need to create **one monitor with three triggers (A, B, and C)** for detecting critical events in Proxmox logs and sending email notifications.

### Trigger A — Critical Keywords

The trigger should fire when the `message` field contains **any** of the following words, case-insensitive:

`error`, `warning`, `alert`, `failed`, `unavailable`, `lost`, `wrong`, `unable`, `invalid`, `failure`

The trigger must not fire if the `message` contains any of these substrings, case-insensitive:

`disabling inotify`, `user config - ignore`, `_put_session`, `PVE ticket`

### Trigger B — Services and Specific Messages

The trigger should fire when the `service` field equals any of the following:

`pvedaemon`, `pveproxy`, `corosync`, `pmxcfs`, `pve-ha-crm`, `pve-ha-lrm`

**AND** the `message` field contains any of the following substrings:

`not exist`, `Stopped corosync`, `Stopped pve-ha-crm`, `Stopped pve-ha-lrm`, `changed from 'online'`, `=> lost_manager_lock`, `=> 'unknown'`, `fence`, `missing resource`

Matching should be case-insensitive.

### Trigger C — Frequency-Based Detection

The trigger should fire when **any one** of the following strings occurs **more than once within a 24-hour period** :

`wait_for_quorum`, `changed from 'online'`

Each string must be counted separately. Therefore:

- `wait_for_quorum` occurs twice → trigger

- `changed from 'online'` occurs twice → trigger

- each occurs once → do not trigger

When **any of the three triggers fires** , the monitor should send an email notification.

Please provide the **complete recommended configuration and Painless trigger conditions for OpenSearch 3.5** , including the correct way to implement the 24-hour frequency condition in Trigger C.

**Configuration** :  
Monitor configuration

```auto
{
  "name": "GRIU.Proxmox service monitoring",
  "type": "monitor",
  "monitor_type": "query_level_monitor",
  "enabled": true,
  "schedule": {
    "period": {
      "unit": "MINUTES",
      "interval": 5
    }
  },
  "inputs": [
    {
      "search": {
        "indices": [
          "griu-*"
        ],
        "query": {
          "size": 100,
          "query": {
            "range": {
              "@timestamp": {
                "from": "now-5m",
                "to": "now",
                "include_lower": true,
                "include_upper": true
              }
            }
          },
          "sort": [
            {
              "@timestamp": {
                "order": "desc"
              }
            }
          ]
        }
      }
    }
  ]
}

```

Trigger A — Critical Keywords:

```auto
Name: Proxmox HA failure
Type: query_level_trigger
Severity: 2

Condition:

def bad = [
    "error",
    "warning",
    "alert",
    "failed",
    "unavailable",
    "lost",
    "wrong",
    "unable",
    "invalid",
    "failure"
];

def ignore = [
    "disabling inotify",
    "user config - ignore",
    "_put_session",
    "PVE ticket"
];

for (def hit : ctx.results[0].hits.hits) {

    String msg = hit._source.message != null
        ? hit._source.message.toString().toLowerCase()
        : "";

    boolean found = false;

    for (def word : bad) {
        if (msg.contains(word)) {
            found = true;
            break;
        }
    }

    if (!found) {
        continue;
    }

    for (def word : ignore) {
        if (msg.contains(word)) {
            found = false;
            break;
        }
    }

    if (found) {
        return true;
    }
}

return false;

```

Trigger B — Proxmox Services and Specific Messages:

```auto
Name: Proxmox general errors
Type: query_level_trigger
Severity: 2

Condition:

def bad = [
    "error",
    "warning",
    "alert",
    "failed",
    "unavailable",
    "lost",
    "wrong",
    "unable",
    "invalid",
    "failure"
];

def ignore = [
    "disabling inotify",
    "user config - ignore",
    "_put_session",
    "PVE ticket"
];

for (def hit : ctx.results[0].hits.hits) {

    String msg = hit._source.message == null
        ? ''
        : hit._source.message.toString().toLowerCase();

    boolean hasBad = false;

    for (def w : bad) {
        if (msg.contains(w)) {
            hasBad = true;
            break;
        }
    }

    if (!hasBad) {
        continue;
    }

    boolean hasIgnore = false;

    for (def x : ignore) {
        if (msg.contains(x)) {
            hasIgnore = true;
            break;
        }
    }

    if (!hasIgnore) {
        return true;
    }
}

return false;

```

Trigger C — Frequency-Based Detection:

```auto
Name: Quorum repeats
Type: query_level_trigger
Severity: 1

Condition:

int quorum = 0;
int online = 0;

for (def hit : ctx.results[0].hits.hits) {

    String msg = hit._source.message == null
        ? ''
        : hit._source.message.toString().toLowerCase();

    if (msg.contains('wait_for_quorum')) {
        quorum++;
    }

    if (msg.contains("changed from 'online'")) {
        online++;
    }
}

return quorum > 1 || online > 1;

```

---

<div class="post-metadata">

**Author:** ![Yanlin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/yanlin/32/10286_2.png) [@Yanlin](https://forum.opensearch.org/u/Yanlin)\
**Post date:** [August 26, 2026, 3:44pm UTC](https://forum.opensearch.org/t/help-configuring-an-opensearch-3-5-alerting-monitor-with-three-triggers/28261/2 "2026-08-26T15:44:45Z")

</div>

Hi @Pan-Vad,

Have you already tired to configure monitors with the describe triggers? If so do you have examples of these monitors? Did you face any issues with your current triggers implementation?

---

<div class="post-metadata">

**Author:** ![Pan-Vad](https://avatars.discourse-cdn.com/v4/letter/p/6de8d8/32.png) [@Pan-Vad](https://forum.opensearch.org/u/Pan-Vad)\
**Post date:** [September 1, 2026, 10:06am UTC](https://forum.opensearch.org/t/help-configuring-an-opensearch-3-5-alerting-monitor-with-three-triggers/28261/3 "2026-09-01T10:06:45Z")

</div>

> [@Yanlin](#):
>
> Have you already tired to configure monitors with the describe triggers? If so do you have examples of these monitors? Did you face any issues with your current triggers implementation?

Yes. I have already configured the monitor and the three triggers, but the current implementation does not work correctly.

The main issue is that the same Proxmox event causes both Trigger A and Trigger B to fire, resulting in two separate email notifications for the same event:

`[CRITICAL] Proxmox HA failure`

and

`[ALERT] Proxmox error detected`

For example, for a single Proxmox event:

`10:09:42 pve1 pvedaemon[905071]: VM 124 qga command failed - VM 124 qga command 'guest-ping' failed - got timeout`

I receive two emails, one from each trigger.

There is also another problem: the event text itself is duplicated in the notification. The message appears approximately as:

`10:09:42 pve1 pvedaemon[905071]: VM 124 qga command failed - VM 124 qga command 'guest-ping' failed - got timeout, 1=VM 124 qga command failed - VM 124 qga command 'guest-ping' failed - got timeout`

So the same log event is effectively included twice in the notification.

I would like to achieve the following behavior:

- One Proxmox log event should result in no more than one email notification.
- Trigger A and Trigger B should not both generate notifications for the same event.
- The email should contain the event message only once.
- Trigger C should independently detect repeated `wait_for_quorum` and `changed from 'online'` events within a 24-hour period, with each string counted separately.
- The monitor should continue to execute every 5 minutes.

Could you please help me correct the monitor and trigger implementation for OpenSearch 3.5 and provide the recommended configuration/Painless scripts to achieve this behavior?

I would especially appreciate clarification on how to prevent two triggers from sending separate notifications for the same log event and how to correctly reference the matching event in the notification instead of simply using the first hit (`hits.hits.0`).

---

<div class="post-metadata">

**Author:** ![Yanlin](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/yanlin/32/10286_2.png) [@Yanlin](https://forum.opensearch.org/u/Yanlin)\
**Post date:** [September 4, 2026, 4:06pm UTC](https://forum.opensearch.org/t/help-configuring-an-opensearch-3-5-alerting-monitor-with-three-triggers/28261/4 "2026-09-04T16:06:28Z")

</div>

Hi @Pan-Vad

Thanks for sharing the configuration.

I reproduced the two-notification behavior in an OpenSearch 3.5 lab. The example event contains `failed`, which matches both Trigger A and Trigger B, so both trigger conditions return `true` and their respective actions are executed. A match in one query-level trigger does not prevent another trigger from matching the same query results, which is consistent with the documented trigger behavior:  
`https://docs.opensearch.org/latest/observing-your-data/alerting/triggers`

The provided Trigger B also does not include the service-specific conditions described earlier and currently uses almost the same keyword logic as Trigger A.

There is also a case-sensitivity issue: the message is converted to lowercase, but the ignore list contains `"PVE ticket"`. This should be `"pve ticket"`, or the ignore values should also be converted to lowercase during comparison.

Trigger C evaluates the results returned by the monitor query. Since the query covers only five minutes, it cannot count events over a 24-hour period. Would you consider using a separate monitor with a `24-hour query window` that still runs every five minutes?

Could you please provide the complete `action message template`, one full matching `_source` document, and `examples of events` that should match only Trigger A and only Trigger B, with any sensitive information redacted?