Kibana email alert - extracting field results

stmx38 · August 27, 2020, 9:20pm

I did a test configuration above using the ‘query’ and ‘chart’. With the ‘query’ I got all required fileds in the messages but with ‘chart’ no, like in your case. It should be investigated why - maybe it is not supported at all.

For you case you can create a test Monitor with the following data:

Query

{
    "size": 1000,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-10h",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "product": {
                            "query": "*Mobile Access*",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "status": {
                            "query": "failure",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

And then create a Trigger with the alert and use ctx iteration in the message.

lehner.angelica · August 28, 2020, 2:53pm

Thanks @stmx38 !!

This actually solved it
I really appreciate your time and help.

Best regards.

lehner.angelica · August 28, 2020, 6:15pm

Hey @stmx38

Could you kindly assist me with writing a query for a monitor that is using more than two terms that one of them is “NOT” X, for example:

That is what I wrote (as far as I understand on how it should work):

{
    "size": 1000,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-1m",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "product": {
                            "query": "*IPS*",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                    {
                    "match_phrase": {
                        "not action": {
                            "query": "*drop*",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "dst": {
                            "query": "10.10.10.10",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

Is that correct?

Also, is there an easy-to-understand documentation on writing queries?

Thanks again!

stmx38 · August 28, 2020, 7:50pm

Your filter in KQL looks like:
product: "*IPS*" AND dst: “172.16.11.1” AND NOT action: “drop”`

In full-text query it may look like this one

{
    "size": 1000,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-1m",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "product": {
                            "query": "*IPS*",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                },
                {
                    "match_phrase": {
                        "dst": {
                            "query": "172.16.11.1",
                            "slop": 0,
                            "zero_terms_query": "NONE",
                            "boost": 1
                        }
                    }
                }
            ],
            "must_not": [
                {
                    "match": {
                        "action": {
                            "query": "*drop*",
                            "prefix_length": 0,
                            "max_expansions": 50,
                            "fuzzy_transpositions": true,
                            "lenient": false,
                            "zero_terms_query": "NONE",
                            "auto_generate_synonyms_phrase_query": true,
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {}
}

In the Discover we can see our current query in the full-text form inside: Inspect --> Request:

Some additional documentation can be found here:

stmx38 · August 28, 2020, 8:08pm

On the creen you provided I see 3 conditions and it looks like it is done in the process of the Monitor definition.

In my Kibana I didn’t find a way to configure more than one condition:

How did you do it?

lehner.angelica · August 29, 2020, 1:58pm

Hey @stmx38,

Thanks!

Regarding the 3 conditions,
I just typed it in the last box, it was working and showed hits.

Regarding full-text query of:
product: “IPS”` AND dst: “x.x.x.x” AND NOT action: “drop”

If I want to write “contains” instead of “is”, what do i need to change in the code above?

Thanks again, much appreciation.

stmx38 · August 29, 2020, 3:40pm

Will this work?

contains

"product": {
    "query": "*IPS*"

is

"dst": {
    "query": "10.10.10.10"

Serge · September 1, 2020, 6:42am

Hi folks.
JFYI: in discovery tab you can setup all the filters you need in Alert. Then you can click “Inspet” - “Request” copy request and modify it.

Mr_Hedgehog · April 16, 2021, 9:10am

Hi there,

I know some time has now past, but I believe the issue with using the visual graph to define the query is that it sets the return size to 0 so there is no result set to iterate over. However there is no option to change this from the graph page.

If you define the query with the visual graph and then switch to the the extraction query you can alter the “size”: 0 to “size”: 1000 and you should see the result set come back in the “hits”.

It is annoying this seems to be the default behavior when using the visual method.

You can see exactly what is being stored when using the visual graph by using ```
GET _opendistro/_alerting/monitors/<monitor_id>

joicyfroit · July 11, 2021, 10:57am

I know even more time has passed. But I just created an account to thank you for this post that ended hours of me pulling my hair out in attempts to solving this puzzle. Thank you so much.

chidex · November 10, 2021, 1:50pm

I have struggled with this alert thing with moustache for months with no clear examples in documentation to guide but stumbling into this saved the months. Thanks a lot stmx38

pitch · December 4, 2021, 9:06pm

this information is absolutely missing in the opensearch documentation
i can’t remember reading about the ctx _source

earlier when getting in touch with opensource projects it was mostly RTFM
sadly, today it looks like it’s RTFC
that’s not userfriendly
what happened
kudos @stmx38

divyank · December 7, 2021, 12:06pm

Hi All and @stmx38,

Thanks for sharing your observations here.

I am able to get values by using the below moustache, but math operations are not performed on it.I don’t have null values in trigger condition.

moustache-

- Trigger: {{ctx.trigger.name}}
- Severity: {{ctx.trigger.severity}}
- Period {{ctx.periodStart}} - {{ctx.periodEnd}}
- Period start: {{ctx.periodStart}}
- Period end: {{ctx.periodEnd}}
- Total logs: {{ctx.track_total_hits}}
-Total Logs: {{ctx.results.0.hits.total.value}}
-Undelivered Logs:{{ctx.results.0.aggregations.when.sum_other_doc_count}}
-Hi : {{ctx.results.0.hits.total.value}} -  {{ctx.results.0.aggregations.when.sum_other_doc_count}}
-Delivered Logs: {{ctx.results.0.hits.total.value}}/{{ctx.results.0.aggregations.when.sum_other_doc_count}}

output-

- Trigger: Undelivered Message Alert
- Severity: 1
- Period 2021-12-07T11:40:43Z - 2021-12-07T11:41:43Z
- Period start: 2021-12-07T11:40:43Z
- Period end: 2021-12-07T11:41:43Z
- Total logs: 
-Total Logs: 277502
-Undelivered Logs:48584
-Hi : 277502 -  48584
-Delivered Logs: 277502/48584

Snap-

stmx38 · December 7, 2021, 5:31pm

Hello @divyank, what do you mean about math operations?

-Delivered Logs: 277502/48584

Should be like this?
-Delivered Logs: 5.7

277502 / 48584 ~ 5.7

divyank · December 7, 2021, 5:37pm

yes @stmx38, like addition,subtraction, etc

stmx38 · December 7, 2021, 6:01pm

Maybe something like this?
{{ ctx.results.0.hits.total.value / ctx.results.0.aggregations.when.sum_other_doc_count }}

divyank · December 7, 2021, 6:41pm

I checked wit it but it’s showing blank values, may be math operations is not possible in moustache’s/

- Hi: {{ ctx.results.0.hits.total.value / ctx.results.0.aggregations.when.sum_other_doc_count }}

Output-
- Hi:

manchou101 · February 10, 2022, 9:44am

@lehner.angelica Hi, I am new to ELK and using opendistro Kibana and trying to sent email alerts using custom webhooks as destination. Could you please help me in configuring a custom webhook using an SMTP server ?

Much appreciated

shaimoh · March 2, 2022, 7:07pm

Hi Angelica,

Good day to you… Im kinda stuck on an issue you once faced during creating alerts in opendistro kibana.

basically, im trying to get the message to come along with the alert when its triggered

{{#ctx.results.0.hits.hits}}
Username: {{_source.user}}
{{/ctx.results.0.hits.hits}}

tried this and did not work…Any advise how you fixed it ? Im using the extraction query to create the alert

sateesh · March 10, 2022, 6:24am

@shaimoh share ur query and response which you used to extract bcz recently i did it if i know i will help you in this…

Topic		Replies	Views
Kibana Alerting Alerting	7	6497	January 5, 2021
Sending a field in a messag Open Source Elasticsearch and Kibana configure , alerting	1	363	October 6, 2022
Include document fields in the message Open Source Elasticsearch and Kibana	1	614	November 6, 2020
Need help to configure the details that are using in custom web hook to send email alerts Alerting troubleshoot	4	1015	February 9, 2022
Email Alert - how do I extract fields in the array inside ctx.results.0.hits.hits.0._source Alerting	2	547	December 22, 2023

Kibana email alert - extracting field results

Related Topics