Help configuring alerting monitor

Plugins Alerting
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): OpenSearch 3.8/OSD 3.8/SLES

Describe the issue:

Please tell me which type of OpenSearch Alerting monitor to choose: “Per query monitor”, “Per bucket monitor”, “Per document monitor”, that will send alerts under the following condition:
A document is critical if the message field contains at least one of the words:
error, warning, alert, failed, unavailable, lost, wrong, unable, invalid, failure.
BUT the document should not be considered critical if message contains at least one of the following strings: disabling inotify, user config - ignore, _put_session, PVE ticket, guest-ping, Cyberprotect. And the document should not be considered critical if the message came from the sshd-session service.

Configuration: Doc

{
“_source”: {
“event”: {},
“@timestamp”: “2026-09-17T08:42:54.000085547Z”,
“host”: “pve3”,
“@version”: “1”,
“tags”: [
“pve”
],
“service”: “sshd-session”,
“type”: “syslog”,
“message”: [
“<38>Sep 17 11:42:53 pve3 sshd-session[844057]: Read error from remote host 1.1.0.1 port 111: Connection reset by peer”,
]
}
}

Relevant Logs or Screenshots:

2

Hi @Pan-Vad,

For an overview of the differences between per-query, per-bucket, and per-document monitors, please refer to the OpenSearch documentation:

You can also find links to the detailed documentation for each monitor type on that page.

I also noticed that some of the conditions you mentioned are similar to those in your previous thread:

Is this related to the same monitoring setup?

Kind regards,

Yanlin.

3

I’d use a per query monitor here. All three of your conditions fit in one bool query, and you get one alert per run instead of one per matching line. A per document monitor works too, but it fires per document and its queries are written in query_string syntax, which gets awkward with a long exclusion list.

Extraction query:

{
  "size": 5,
  "query": {
    "bool": {
      "filter": [
        { "range": { "@timestamp": { "gte": "{{period_end}}||-5m", "lte": "{{period_end}}", "format": "epoch_millis" } } }
      ],
      "should": [
        { "match": { "message": "error warning alert failed unavailable lost wrong unable invalid failure" } }
      ],
      "minimum_should_match": 1,
      "must_not": [
        { "match_phrase": { "message": "disabling inotify" } },
        { "match_phrase": { "message": "user config - ignore" } },
        { "match_phrase": { "message": "_put_session" } },
        { "match_phrase": { "message": "PVE ticket" } },
        { "match_phrase": { "message": "guest-ping" } },
        { "match_phrase": { "message": "Cyberprotect" } },
        { "term": { "service.keyword": "sshd-session" } }
      ]
    }
  }
}

Trigger condition: ctx.results[0].hits.total.value > 0

Two things to check against your mapping. If service is mapped as keyword rather than text, use “service” instead of “service.keyword”. And match on the standard analyzer does not stem, so “fail” or “errors” will not match unless you list them too. Keep the range window equal to the monitor interval so lines aren’t skipped or counted twice.

4

Hello,

I need the system to send a notification for every new matching document, rather than only once while the condition remains true.

For example, I sent three test log entries with a 5-second delay between each command:

logger -t 'test_pve1' 'qwerty - failed - QWERTY - 1'
logger -t 'test_pve1' 'qwerty - invalid - QWERTY - 2'
logger -t 'test_pve1' 'qwerty - lost - QWERTY - 3'

with a 5-second delay between each command. As a result, I received only one email, whereas I expected three emails.

Please advise on how to configure this so that each new matching document triggers a separate notification.

Thank you.

5

I read the documentation, but unfortunately it did not clarify my issue. I created a new thread because I changed the monitor condition and updated the OpenSearch version.

6

Do I understand correctly that the “Per document monitors” monitor type cannot work with a dynamic index?

7

Hi @Pan-Vad

Thanks for the clarification.

For generating a separate alert for each matching document, a Per document monitor would be the appropriate monitor type. I tested this on OpenSearch 3.8.0 and confirmed that three matching documents in a fixed index generated three separate alerts.

However, Per document monitors do not support wildcard index patterns in OpenSearch 3.8.0. When I tried creating a monitor using a wildcard pattern, I received the following error:
Validation Failed: 1: Index patterns are not supported for doc level monitors.

I also tested using an index alias as an alternative. It worked with a single index, but with multiple indices, it detected new documents in the write index but not in the non-write index. After rollover, the existing monitor did not detect matching documents in the new write index. These were the results observed in my OpenSearch 3.8.0 tests.

Kind regards,
Yanlin