Include scripted field in Kibana monitor extraction query

Plugins Alerting
1

Hi,
I’m defining a monitor for Kibana alerting and I’d like to include the service scripted field (which I’ve prepared) in the _source (or any other accessible field). I’d like to use that information in the Slack notification message, by doing something like {{_source.service}}. Currently, I don’t see a way to do it.

Defined extraction query

{
    "query": {
        "bool": {
            "must": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "now-30m",
                            "to": null,
                            "include_lower": true,
                            "include_upper": true,
                            "boost": 1
                        }
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "_source": {
        "includes": [
            "country",
            "@timestamp",
            "timestamp",
            "service",
            "environment",
            "function_name",
            "level"
        ],
        "excludes": []
    }
}

Hits from extraction query response

"hits": {
        "hits": [
            {
                "_index": "some-index",
                "_type": "some-information",
                "_source": {
                    "country": "pl",
                    "environment": "prod",
                    "@timestamp": "2020-03-04T12:34:39.581Z",
                    "level": "ERROR",
                    "function_name": "dev.opendistrocommunity.discuss.problem",
                    "timestamp": "2020-03-04 12:34:39,581"
                },
                "_id": "3530...38",
                "_score": 10.790063
            }
        ],
        "total": 1,
        "max_score": 10.790063
    }

JSON information about log

{
  "_index": "...",
  "_type": "...",
  "_id": "3530...38",
  "_version": 1,
  "_score": null,
  "_source": {
    "correlation_id": "...",
    "request_id": "...",
    "message": "Internal Server Error",
    "timestamp": "2020-03-04 10:43:26,751",
    "level": "ERROR",
    "function_name": "dev.opendistrocommunity.discuss.problem",
    "thread": "...",
    "environment": "prod",
    "country": "pl",
    "@id": "3530...38",
    "@timestamp": "2020-03-04T10:43:26.751Z",
    "@message": "...",
    "@owner": "...",
    "@log_group": "...",
    "@log_stream": "..."
  },
  "fields": {
    "service": [
      "_____INFORMATION-I-NEED-IS-HERE______"
    ],
    "@timestamp": [
      "2020-03-04T10:43:26.751Z"
    ]
  },
  "highlight": {
    "level": [
      "@kibana-highlighted-field@ERROR@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1583318606751
  ]
}

I have defined Monitor by Define using extraction query and general index *****, for which the mentioned scripted field is defined.

Screenshot from 2020-03-04 14-29-19
Screenshot from 2020-03-04 14-29-19723×326 14.9 KB

I’d appreciate some help.

2

Hi @beam022,

The response of the query should be available on {{ctx.results.0}}.

3

Thanks for replying and you’re right, but it doesn’t bring me closer to a solution. I know how to access results - the problem is in putting a scripted field value in these results.

4

So it’s rather about what I have to put in extraction query to have that field available for further processing.

5

Hey @beam022 did you ever work it out?
All I want to do is truncate a field in the alert.
Can’t do it in the dumb Mustache, so looking to do it as a projection on the query

6

@jnerm No, sorry, haven’t figured it out and haven’t received much help on this. If you do, please let me know how.

1 Like
7

Hi there - I think I am having a similar issue to yourself. Did you manage to get anywhere with this? Thanks.

8

@Mr_Hedgehog No, sorry. I have given up on this.

9

If I understand the question correctly, if the service field is just a constant, you can define it using script fields as follows:

GET example_query/_search
{
  "query": {
    "match_all": {}
  },
  "_source": {
    "includes": [
      "field1",
      "field2",
      "field3"
    ],
    "excludes": []
  },
  "script_fields": {
    "service": {
      "script": {
        "lang": "painless",
        "source": "'my_service_name'"
      }
    }
  }
}

You also can perform String concatenation and have access to the doc contents in the script in case you want the service name to be something more dynamic. For example:

"script_fields": {
    "service": {
      "script": {
        "lang": "painless",
        "source": "'my_service_name' + '_' + doc['environment'].value"
      }
    }
  }

Then you can access these fields similar to how would access _source while iterating over the search hits. For example:

{{#ctx.results.0.hits.hits}}
  {{_source.country}}
  {{fields.service.0}}
{{/ctx.results.0.hits.hits}}

When I tested out the scripted field, the response was an array, hence the 0 index in the example above.

Hope that helps.