How to get array value in opensearch dashboard

OpenSearch Dashboards
1

Hi,
We are using opensearch dashboards with the version of 2.3.0.
We can see there are different values for the field “partition” for the “XYZ” operation logs (logtype is same and operation is same).
loglines are:
(1). 2023-01-05T20:46:21.36348538Z stdout F 2023-01-05 20:46:21.363 [INFO] - {“logtype”:“ABC”,“operation”:“XYZ”,“partition”:,“topic”:“INT_XXXX_TD”}

(2). 2023-01-05T20:38:26.998223061Z stdout F 2023-01-05 20:38:26.998 [INFO] - {“logtype”:“ABC”,“operation”:“XYZ”,“partition”:[{“Topic”:“INT_XXXX_RETRY”,“Partition”:1,“Offset”:-1111,“Metadata”:null,“Error”:null},{“Topic”:“INT_XXXX_RETRY”,“Partition”:0,“Offset”:-1111,“Metadata”:null,“Error”:null}],“topic”:“INT_XXXX_RETRY”}

(3) 2023-01-05T20:46:21.36312474Z stdout F 2023-01-05 20:46:21.362 [INFO] - {“logtype”:“ABC”,“operation”:“XYZ”,“partition”:[{“Topic”:“INT_XXXX_TD”,“Partition”:0,“Offset”:-1111,“Metadata”:null,“Error”:null}],“topic”:“INT_XXXX_TD”}

In above 2 logs the “partition” value is not constant some times it is “null” and sometimes it is “Array”. I need to drop “partition” value is “null” logs and only i need to get remaining logs (i.e log number 2 &3).Could you please guide me how to get “partition” as a field in kibana and which condition i need to mentioned in logstash config.

Configuration in Logstash:-

if(“assigned-partition” in [kafka][operation] or “revoked-partition” in [kafka][operation])
{
mutate {
rename => [“[kafka][topic]”, “[kafka][topic][name]” ]
add_field => { “[kafka][messagetype]” => “consumerEvents” }

2

You can use exists query to get the array value, like this:

GET test1/_search
{
  "query": {
    "exists": {
      "field": "partition"
    }
  }
}

,and use drop filter in Logstash to drop the log whose partition field is null.