Alert condition for Per Bucket Monitor

OpenSearch
, , ,
1

Hi,
Following is my extraction query:

{
    "size": 0,
  "query": {
    "terms": {
      "eventId": ["4624", "4625", "4740"]
    }
  },
  "aggs": {
    "by_agent": {
      "terms": {
        "field": "agent_id.keyword",
        "size": 100
      },
      "aggs": {
        "by_eventId": {
          "terms": {
            "field": "eventId.keyword",
            "size": 100
          }
        }
      }
    }
  }
}

and I am getting result:

{
    "_shards": {
        "total": 1,
        "failed": 0,
        "successful": 1,
        "skipped": 0
    },
    "hits": {
        "hits": [],
        "total": {
            "value": 15,
            "relation": "eq"
        },
        "max_score": null
    },
    "took": 5,
    "timed_out": false,
    "aggregations": {
        "by_agent": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
                {
                    "doc_count": 9,
                    "by_eventId": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                            {
                                "doc_count": 5,
                                "key": "4740"
                            },
                            {
                                "doc_count": 2,
                                "key": "4624"
                            },
                            {
                                "doc_count": 2,
                                "key": "4625"
                            }
                        ]
                    },
                    "key": "000"
                },
                {
                    "doc_count": 6,
                    "by_eventId": {
                        "doc_count_error_upper_bound": 0,
                        "sum_other_doc_count": 0,
                        "buckets": [
                            {
                                "doc_count": 3,
                                "key": "4740"
                            },
                            {
                                "doc_count": 2,
                                "key": "4625"
                            },
                            {
                                "doc_count": 1,
                                "key": "4624"
                            }
                        ]
                    },
                    "key": "007"
                }
            ]
        }
    }
}

I want to trigger an alert for each document (with 000 and 007)
if (key==4740 && doc_count > 1) && (key==4625&& doc_count > 2) && (key==4624&& doc_count > 3)
return true // raise an alert

I tried various ways to achieve this, but nothing is working.
I always get “Empty list doesn’t contain element at index 0.”

Can someone please help me for this?

2

@binita Not sure if this will work for your use case, but see the following example that I got working.

Assuming the index looks like this:

"hits": [
      {
        "_index": "demo-alert",
        "_id": "v513xZcBYXdLCGtLYl0I",
        "_score": 1,
        "_source": {
          "agent_id": "000",
          "eventId": "4740",
          "timestamp": "2025-07-01T12:00:00"
        }
      },

You should be able to create a monitor as follows:

POST _plugins/_alerting/monitors
{
  "name": "Test_Monitor",
  "monitor_type": "query_level_monitor",
  "enabled": true,
  "schedule": {
    "period": {
      "interval": 1,
      "unit": "MINUTES"
    }
  },
  "inputs": [
    {
      "search": {
        "indices": [
          "demo-alert"
        ],
        "query": {
          "size": 0,
          "aggs": {
            "agents": {
              "terms": {
                "field": "agent_id",
                "size": 100
              },
              "aggs": {
                "event_4740": {
                  "filter": {
                    "term": {
                      "eventId": "4740"
                    }
                  }
                },
                "event_4625": {
                  "filter": {
                    "term": {
                      "eventId": "4625"
                    }
                  }
                },
                "event_4624": {
                  "filter": {
                    "term": {
                      "eventId": "4624"
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  ],
  "triggers": [
    {
      "query_level_trigger": {
        "name": "All Events Threshold Exceeded",
        "severity": "1",
        "condition": {
          "script": {
            "source": """
              def agents = ctx.results[0].aggregations.agents.buckets;
              for (agent in agents) {
                def e4740 = agent.event_4740.doc_count;
                def e4625 = agent.event_4625.doc_count;
                def e4624 = agent.event_4624.doc_count;
                if (e4740 > 1 && e4625 > 2 && e4624 > 3) {
                  return true;
                }
              }
              return false;
            """,
            "lang": "painless"
          }
        },
        "actions": [
          {
            "name": "Slack alert",
            "destination_id": "dAd5......",
            "message_template": {
              "source": """
Agent event thresholds exceeded:

{{#ctx.results.0.aggregations.agents.buckets}}
- Agent {{key}} → 4740={{event_4740.doc_count}}, 4625={{event_4625.doc_count}}, 4624={{event_4624.doc_count}}
{{/ctx.results.0.aggregations.agents.buckets}}
""",
              "lang": "mustache"
            }
          }
        ]
      }
    }
  ]
}
3

Hello @Anthony,

For the trigger part,
If multiple buckets have matching criteria, the code will also exit at the first true condition and send an alert.

I prefer 1 alert for 1 matching criterion. Here, it is not the case.

For notification, I would like to send a notification only for the agent_id (key) that matched the condition.

Can you please show me the way?

4

@binita I would recommend to use transform job to run aggregation and store the results in a separate index, using something like this:

PUT _plugins/_transform/agent_event_summary_transform
{
  "transform": {
    "enabled": true,
    "continuous": true,
    "schedule": {
      "interval": {
        "period": 1,
        "unit": "Minutes",
        "start_time": 1602100553
      }
    },
    "description": "Count specific event types per agent_id",
    "source_index": "main-index",
    "target_index": "supporting-index",
    "data_selection_query": {
      "match_all": {}
    },
    "page_size": 1000,
    "groups": [
      {
        "terms": {
          "source_field": "agent_id",
          "target_field": "agent_id"
        }
      }
    ],
    "aggregations": {
      "event_4740": {
        "scripted_metric": {
          "init_script": "state.count = 0;",
          "map_script": """
            if (doc.containsKey('eventId') && doc['eventId'].size() != 0 && doc['eventId'].value == '4740') {
              state.count += 1;
            }
          """,
          "combine_script": "return state.count;",
          "reduce_script": "return states.stream().mapToInt(s -> s).sum();"
        }
      },
      "event_4625": {
        "scripted_metric": {
          "init_script": "state.count = 0;",
          "map_script": """
            if (doc.containsKey('eventId') && doc['eventId'].size() != 0 && doc['eventId'].value == '4625') {
              state.count += 1;
            }
          """,
          "combine_script": "return state.count;",
          "reduce_script": "return states.stream().mapToInt(s -> s).sum();"
        }
      },
      "event_4624": {
        "scripted_metric": {
          "init_script": "state.count = 0;",
          "map_script": """
            if (doc.containsKey('eventId') && doc['eventId'].size() != 0 && doc['eventId'].value == '4624') {
              state.count += 1;
            }
          """,
          "combine_script": "return state.count;",
          "reduce_script": "return states.stream().mapToInt(s -> s).sum();"
        }
      }
    }
  }
}

This would produce an index with the following results:

"hits": [
      {
        "_index": "supporting-index",
        "_id": "8Bv7g--9oHL0R37RppTBAg",
        "_score": 1,
        "_source": {
          "transform._id": "agent_event_summary_transform22",
          "_doc_count": 40,
          "transform._doc_count": 40,
          "agent_id": "000",
          "event_4625": 10,
          "event_4624": 10,
          "event_4740": 20
        }
      },

You can then create alerts using this new index. Something similar to the following:

POST _plugins/_alerting/monitors
{
  "name": "Agent Event Thresholds Monitor",
  "monitor_type": "query_level_monitor",
  "enabled": true,
  "schedule": {
    "period": {
      "interval": 1,
      "unit": "MINUTES"
    }
  },
  "inputs": [
    {
      "search": {
        "indices": ["supporting-index"],
        "query": {
          "size": 0,
          "query": {
            "bool": {
              "must": [
                { "range": { "event_4740": { "gt": 1 } } },
                { "range": { "event_4625": { "gt": 2 } } },
                { "range": { "event_4624": { "gt": 3 } } }
              ]
            }
          }
        }
      }
    }
  ],
  "triggers": [
    {
      "query_level_trigger": {
        "name": "Agent Threshold Trigger",
        "severity": "1",
        "condition": {
          "script": {
            "source": "return ctx.results[0].hits.total.value > 0;",
            "lang": "painless"
          }
        },
        "actions": [
          {
            "name": "Send Slack Alert",
            "destination_id": "3DNUz.....",
            "message_template": {
  "source": "One or more agents exceeded thresholds in event counts.",
  "lang": "mustache"
}
          }
        ]
      }
    }
  ]
}
5

Hello Anthony,
Thank you very much for the reply. We are trying to implement your suggested solution and will update you here.

Let me give you some context for these exercises.
We have a requirement to implement a SIEM rule for one of the clients. Basic overview of the rule is,

We need to monitor logs every 10 mins and raise an alert with appropriate details if 3 of the following conditions match.
(1) count (event_id = XXXX) > 2
(2) count (firewall logs_somelogic) > 10
(3) count (switch logs_somelogic) > 10

Ideally, we would like to use detectors for these, as findings make things straightforward, but detectors/rules do not provide “group by” functionality in the current version of OpenSearch (Please correct me if this is not true).

So, we decided to create a composite monitor with different child monitors to achieve this scenario. When we struggled with the trigger part, this post came into existence!

In the current stage, we are working on the child monitors individually to set the logic. But we will have to bubble up these child alert details to the parent monitor to send an appropriate message(notification) to the Admin.