Opensearch - Drop processor if numerical field value is less than

OpenSearch
,
1

Version: 2.11.1

I’m trying to drop events going into my Index in Opensearch where if the value in the “delay” field is less than 500 drop it.

I’ve tried doing this with the following code in my ingest pipeline for that index but doesn’t seem to be doing what I want it to and not giving any errors to suggest there is an issue.

“drop”: {
“if”: “ctx.delay <= ‘500’”
}

I have tried without putting single quotes around the 500 number and when I remove ctx it fails the pipeline request.

The delay field is a field I have managed to extract from the message field via a grok expression and I’ve set this to numerical in both the grok expression and via the index mappings and it shows up in Opensearch as it’s own field and and is shown as a numerical field.

However, all the examples I see regarding the drop processor all seemed to be based on string value and are always done like if the value contains or if the value is ==.

I was wondering if anyone has done this before to see if it’s possible to use the drop processor this way and if so how did you manage to get it to work.

Thanks.

2

Hey @MrAssassins

what kind of log shipper are you using? and how you ingesting those logs?

I use Logstash, so I have my grok pattern set up to drop what I dont need prior to send it to Opensearch. Just an idea.

3

Hello @Gsmitt ,

Im using Filebeat to ship the logs and i have an opensearch ingest pipeline file that automatically creates the index with a date stamp where my grok pattern is to try and filter out the message data where the “delay” value i want to filter out exists.

My understanding was that using grok patterns to drop data meant that the event still goes into Opensearch but that data from the event itself is dropped. In my case i wanted to stop the event being ingested if the delay value is less than or equal to 500. Wasnt sure if that was possible with a grok expression?

But if so i will definitely look into this :slight_smile:

4

Can you post you GROK?

5

%{DATA:date} %{DATA:pipe} %{DATA:time} %{DATA:pipe} %{NUMBER:delay} %{DATA:pipe} %{IP:c_ip} %{DATA:pipe} %{IP:s_ip} %{DATA:pipe} %{NUMBER:port} %{DATA:pipe} %{WORD:cs_method} %{DATA:pipe} %{NUMBER:cs_uri} %{DATA:pipe} %{DATA:cs_uri_stem} %{DATA:pipe} %{DATA:cs_uri_query} %{DATA:pipe} %{DATA:sc_status} %{DATA:pipe} %{DATA:cs_user_agent} %{DATA:pipe} %{DATA:cs_referer} %{DATA:pipe} %{DATA:cs_x_api_key}

The data was seperated via a | in the log file and had to define that otherwise the grok expression wouldn’t work how I needed it to but it’s the %{NUMBER:delay} field I’m trying to filter out to drop events for if that value is 500 or less

6

hey

Im assuming your using API to set your GROK to drop events?

Filebeat can be configured something like that.

processors:
  - drop_event:
      when:
        condition

Or something like this.

processors:
  - drop_fields:
      when:
        condition
      fields: ["field1", "field2", ...]
      ignore_missing: false
7

I did see this but wasn’t sure if the drop_event processor was able to define a grok pattern and then use a condition to drop the event based on a field defined in a grok pattern?