How to configure "per query monitor" to trigger alerts according to each error logs on query result

DevOps
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Opensearch 2.15

Describe the issue:
I have a monitor for error logs level with multiple index in 1 Cluster, I’m using the “Per query monitor” type with interval of 1 minutes, the query search for “Error” and trigger alarm if there is errors during that 1 minute.
→ The issue is that if during that 1 minute, we have multiple errors, just only 1 alert is generated and push to our team, we want to set up that each error log will trigger an alert for it.

Configuration:

  • Per query monitor
  • interval: 1 minute
  • Select data: multiple indexes from local Cluster
  • Query:
    "
    {
    “size”: 10,
    “query”: {
    “bool”: {
    “must”: [
    {
    “match_phrase”: {
    “level”: {
    “query”: “Error”,
    “slop”: 0,
    “zero_terms_query”: “NONE”,
    “boost”: 1
    }
    }
    }
    ],
    “filter”: [
    {
    “range”: {
    “@timestamp”: {
    “from”: “{{period_end}}||-1m”,
    “to”: “{{period_end}}”,
    “include_lower”: true,
    “include_upper”: true,
    “format”: “epoch_millis”,
    “boost”: 1
    }
    }
    }
    ],
    “adjust_pure_negative”: true,
    “boost”: 1
    }
    },
    “_source”: {
    “includes”: [
    “@timestamp”,
    “message”,
    “level”
    ],
    “excludes”:
    },
    “sort”: [
    {
    “@timestamp”: {
    “order”: “desc”
    }
    }
    ]
    }
    "
  • Trigger condition: ctx.results[0].hits.total.value > 0
  • Message:
    "
    {
    “message”: “ALARM: {{ctx.results.0.hits.hits.0._index}}”,
    “description”: “ERROR LOG: {{ctx.results.0.hits.hits.0._source.message}}”,
    “tags”: [“Staging”],
    “alias”: “{{ctx.results.0.hits.hits.0._index}}- {{ctx.results.0.hits.hits.0._source.message}}”,
    “priority”: “P3”,
    “details”: {
    “AWSAccountId”: “905417996969”
    }
    }
    "

Relevant Logs or Screenshots:

2

Hi @andrew_flying,

If my understanding is correct, per query type treats the query as a whole unit and evaluates aggregated conditions like ctx.results[0].hits.total.value > 0, even though your query pulls multiple logs, the trigger logic references only the first hit.

Have you considered using per document type?

Best,
mj

3

Hi @Mantas ,

Thank you for your response.

If I have n indexes to query like: a*, b*, c*,… n* in my previous per query monitor and I want to change to use per document monitor, I need to create n document monitors, right?

I’m new to Opensearch, could you give me example config of query, trigger and message that use to alert when each time index a* got error log.

Thank so much,
Andrew,

4

Per-query monitors only evaluate the query once per interval, so only one alert is triggered per run. To get alerts for each error, use bucket-level monitors with a grouping field (like message or a unique ID) to trigger individual alerts for each log entry.

5

If I would need to send the error logs to each alert, How can I configure it in “Bucket monitor” type?