Opensearch false alert

I received alert at 06/02/22 2:44PM but when I manually query the log there is no hit.
Here is my extraction query

{
    "size": 5,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-2m",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "query_string": {
                        "query": "*@error*",
                        "fields": [],
                        "type": "best_fields",
                        "default_operator": "or",
                        "max_determinized_states": 10000,
                        "enable_position_increments": true,
                        "fuzziness": "AUTO",
                        "fuzzy_prefix_length": 0,
                        "fuzzy_max_expansions": 50,
                        "phrase_slop": 0,
                        "escape": false,
                        "auto_generate_synonyms_phrase_query": true,
                        "fuzzy_transpositions": false,
                        "boost": 1
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

I manually ran the query from the monitor editing page (with time including the alert triggered period) but I got no hit.


By the way our cluster is recently under great pressure, the Opensearch log indexing rate is smaller than Logstash output rate so there was log delay. Is there any relationship between?

Hi @gen_yang,

Was the created Alert in error state or did it go from active to completed? You can check the alert history index if it’s enabled.

There is a ctx.error that contains error info when an execution runs into anything but if that wasn’t included in the message that was sent out, it might look like a false positive. Just want to make sure we eliminate that as a possibility.