Kibana email alert - extracting field results

If i type:

{{ctx.results.0.hits.total.value}}

On the message pane, i get “73”.

It means that Monitor return some data. And now we should iterate over it:

{{#ctx.results.0.hits.hits}}
All data: {{_source}}
Username: {{_source.user}}
{{/ctx.results.0.hits.hits}}

This is the problem, It doesn’t work :confused:

Message preview is blank.

Please do some tests without iteration:

-- Without iteration:
All Data: {{ctx.results.0.hits.hits.0._source}}
Timestamp: {{ctx.results.0.hits.hits.0._source.@timestamp}}
Scope: {{ctx.results.0.hits.hits.0._source.Scope}}
Message: {{ctx.results.0.hits.hits.0._source.Message}}

No results…

Just to clarify, I do have results on “Discovery” and can make alerts by specific terms (which all work).

Just prepared a short guide how to create an alarm based on the query. You can easy test it or in case of need adapt to your case.

1. Create an index

The index with the prefix vpn-log-test will be created. You can change prefix in the scrips variable

# Variables
elasticsearch_url=http://localhost:9200
date=$(date +%Y-%m-%d)
index_name=vpn-log-test-$date
index_type=default

users="Alice Bob"
error="VPN connection failed"

# Log to the Elasticsearch
for user in $users; do
  time=$(date +%Y-%m-%d'T'%H:%M:%S.%3N)

  curl -H "Content-Type: application/json" \
     -XPOST "$elasticsearch_url/$index_name/$index_type" \
     -d "{\"Time\":\"$time\", \"User\":\"$user\", \"Error\":\"$error\"}"
     sleep 2
done
2. Create an Index Pattern
Kibana --> Management --> Index Patterns --> Create index pattern:

Index pattern: vpn-log-test*
Time Filter field name: time

--> Create index pattern
3. Discover the data

4. Create a Monitor based on the query
{
    "size": 1000,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "Time": {
                            "from": "{{period_end}}||-10h",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "Error": {
                            "query": "*VPN connection failed*",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

5. Create a Trigger for the created Monitor

Message

Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.
- Trigger: {{ctx.trigger.name}}
- Severity: {{ctx.trigger.severity}}
- Period {{ctx.periodStart}} - {{ctx.periodEnd}}
- Count: {{ctx.results.0.hits.total.value}}

{{#ctx.results.0.hits.hits}}
{{_source.Time}} - {{_source.User}} - {{_source.Error}}
{{/ctx.results.0.hits.hits}}

6. Check Alarm

Question related to the alarm based on the chart still need to be investigated.

4 Likes

Found an interesting information in the forum thread:

You could click the info button near the Trigger condition editor area in Define trigger page. It will show you directly with JSON response of what’s available under the “ctx” variable.
I can see such button only in case of the query, not chart.

Wow, Thank you for elaborating.

I think I kind of did all.
I deleted the old VPN index and created a new index pattern, called “checkpoint*”. (our firewall)
The data get indexed successfully on the “discovery” node, as can be seen here (I filtered the results):

I created a monitor, based on the query for failed vpn login (shows hits), as can be seen here:

After that, I created a trigger (which also shows hits):

Created an Alarm \ Action:

Everything seems to be configured correctly.
Alerts are being sent by email without further issues everytime there’s a failed login (usually on the monitor I’m using 1 minutes and not 1 day as the example above).

The basic CTX commands are working without any issues but when I’m trying to use one of your suggestions above (with or without iteration) nothing seems to be displayed.

I’m not an experienced programmer, am i missing something during the process?

Thanks again!
Your help is much appreciated.

I did a test configuration above using the ‘query’ and ‘chart’. With the ‘query’ I got all required fileds in the messages but with ‘chart’ no, like in your case. It should be investigated why - maybe it is not supported at all.

For you case you can create a test Monitor with the following data:

Query
{
    "size": 1000,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-10h",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "product": {
                            "query": "*Mobile Access*",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "status": {
                            "query": "failure",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

And then create a Trigger with the alert and use ctx iteration in the message.

2 Likes

Thanks @stmx38 !!

This actually solved it :slight_smile:
I really appreciate your time and help.

Best regards.

1 Like

Hey @stmx38

Could you kindly assist me with writing a query for a monitor that is using more than two terms that one of them is “NOT” X, for example:

That is what I wrote (as far as I understand on how it should work):

{
    "size": 1000,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-1m",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "product": {
                            "query": "*IPS*",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                    {
                    "match_phrase": {
                        "not action": {
                            "query": "*drop*",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "dst": {
                            "query": "10.10.10.10",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

Is that correct?

Also, is there an easy-to-understand documentation on writing queries?

Thanks again!

1 Like

Your filter in KQL looks like:
product: "*IPS*" AND dst: “172.16.11.1” AND NOT action: “drop”`

In full-text query it may look like this one
{
    "size": 1000,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-1m",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "product": {
                            "query": "*IPS*",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "dst": {
                            "query": "172.16.11.1",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                }
            ],
            "must_not": [
                {
                    "match": {
                        "action": {
                            "query": "*drop*",
                            "prefix_length": 0,
                            "max_expansions": 50,
                            "fuzzy_transpositions": true,
                            "lenient": false,
                            "zero_terms_query": "NONE",
                            "auto_generate_synonyms_phrase_query": true,
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

In the Discover we can see our current query in the full-text form inside: Inspect --> Request:

Some additional documentation can be found here:

  1. Full-text queries
  2. Full text queries

On the creen you provided I see 3 conditions and it looks like it is done in the process of the Monitor definition.

In my Kibana I didn’t find a way to configure more than one condition:

How did you do it?

Hey @stmx38,

Thanks!

Regarding the 3 conditions,
I just typed it in the last box, it was working and showed hits.

Regarding full-text query of:
product: “IPS”` AND dst: “x.x.x.x” AND NOT action: “drop”

If I want to write “contains” instead of “is”, what do i need to change in the code above?

Thanks again, much appreciation.

Will this work?

contains

"product": {
    "query": "*IPS*"

is

"dst": {
    "query": "10.10.10.10"

Hi folks.
JFYI: in discovery tab you can setup all the filters you need in Alert. Then you can click “Inspet” - “Request” copy request and modify it.

2 Likes

Hi there,

I know some time has now past, but I believe the issue with using the visual graph to define the query is that it sets the return size to 0 so there is no result set to iterate over. However there is no option to change this from the graph page.

If you define the query with the visual graph and then switch to the the extraction query you can alter the “size”: 0 to “size”: 1000 and you should see the result set come back in the “hits”.

It is annoying this seems to be the default behavior when using the visual method.

You can see exactly what is being stored when using the visual graph by using ```
GET _opendistro/_alerting/monitors/<monitor_id>

I know even more time has passed. But I just created an account to thank you for this post that ended hours of me pulling my hair out in attempts to solving this puzzle. Thank you so much.

2 Likes

I have struggled with this alert thing with moustache for months with no clear examples in documentation to guide but stumbling into this saved the months. Thanks a lot stmx38

1 Like

this information is absolutely missing in the opensearch documentation
i can’t remember reading about the ctx _source

earlier when getting in touch with opensource projects it was mostly RTFM
sadly, today it looks like it’s RTFC
that’s not userfriendly
what happened
kudos @stmx38

1 Like