Opensearch false alert

Plugins Alerting
1

I received alert at 06/02/22 2:44PM but when I manually query the log there is no hit.
Here is my extraction query

{
    "size": 5,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-2m",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "query_string": {
                        "query": "*@error*",
                        "fields": [],
                        "type": "best_fields",
                        "default_operator": "or",
                        "max_determinized_states": 10000,
                        "enable_position_increments": true,
                        "fuzziness": "AUTO",
                        "fuzzy_prefix_length": 0,
                        "fuzzy_max_expansions": 50,
                        "phrase_slop": 0,
                        "escape": false,
                        "auto_generate_synonyms_phrase_query": true,
                        "fuzzy_transpositions": false,
                        "boost": 1
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

I manually ran the query from the monitor editing page (with time including the alert triggered period) but I got no hit.

image
image3352×1546 367 KB

By the way our cluster is recently under great pressure, the Opensearch log indexing rate is smaller than Logstash output rate so there was log delay. Is there any relationship between?

2

Hi @gen_yang,

Was the created Alert in error state or did it go from active to completed? You can check the alert history index if it’s enabled.

There is a ctx.error that contains error info when an execution runs into anything but if that wasn’t included in the message that was sent out, it might look like a false positive. Just want to make sure we eliminate that as a possibility.

3

I have exactly the same issue.
Numerous false alerts.
What I found that in ‘.opendistro-alerting-alert-history-2022.11.24-000302’ ‘index’ there is something may have been treated as explanation:

          "monitor_name": "zabbix.site1.company.com",
          "monitor_user": null,
          "trigger_id": "iFf7s3oBmcdqlGU9CLca",
          "trigger_name": "out_of_memory",
          "state": "COMPLETED",
          "error_message": null,
          "alert_history": [
            {
              "timestamp": 1671454961520,
              "message": "Failed fetching inputs:\nFailed to execute phase [fetch], ; org.opensearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<reduce_aggs>] would be [8495916310/7.9gb], which is larger than the limit of [8160437862/7.5gb], real usage: [8495916272/7.9gb], new bytes reserved: [38/38b], usages [request=113/113b, fielddata=345165/337kb, in_flight_requests=0/0b, accounting=559224372/533.3mb]"
            }
          ],

It seems like monitoring failed while it has to run on large ‘index’ (more exact, ‘index’ set as ‘index-name-*’)

Could anybody point to solution or workaround for this?