Scripted fields filtering in visualizations

OpenDistro
1

Hello,
I’m trying to create visualizations for AWS WAF logs. So far the dashboard is working great, and most of the visualizations are working correctly.
But some of the visualizations which rely on scripted fields, are failing to be filtered.
By being filtered i mean the little + button inside the visualization (when filtering for a specific value).

So basically what happens when you click on that is that the script crashes and simply doesn’t display anything.

Below is the response with the reason why it failed

image
image696×586 48.4 KB

According the response the issue is within the compare() function, but its not a function that was written by me, i believe its the built-in function for the value filtering within the visualization.

The scripted field is User-Agent, and the script is as follows :
def header_name=‘’; for(int i=0; i<params._source.httpRequest.headers.length; i++) { header_name = params._source.httpRequest.headers[i].name.toLowerCase(); if (header_name == “user-agent”) { return params._source.httpRequest.headers[i].value } }
(I have to loop through the values because i cant reference them directly)

Does anyone has any idea why this happens?
*I’ve tried this on those versions of Kibana (6.7,6.8) and Opensearch 1.0 which are hosted on AWS Opensearch service.

Thanks in advance!

2

Hey Alexeysh,

I’m by no means an expert here but what is interesting about this error is there are 23 successes and 32 failures. This to me makes me believe that there is some sort of value errors with some of the records or something along those lines. Can you identify individual documents that are succeeding or failing?

3

Hey dtaivpp,

Thanks for replying!
Thats a good question you’re asking, Ive compared the indexes that failed and succeeded and sometimes the index mapping changes. So i scoped the search to a single day where the mapping is the same, and it still gave me the same error.
But this time it was 25 successful and only 1 shard failed.

image
image885×791 34.4 KB

I believe this is the same issue.

4

In case someone else stumbles across this, while scripted fields using params._source will work with search and aggregation, if a filter is applied to those scripted fields, they will fail. This is because params._source was removed from filters some time ago. Here, Kibana/OpenSearch Dashboards, while using the definition of the scripted field in the filter, come across params._source and the script fails due to it being null.