Logstash-output-opensearch not using my timestamp

OpenSearch
1

Hi all. I’m using the logstash-output-opensearch. It’s working, but I can’t get it to use my log timestamp as @timestamp.

This is what I want it to use, direct from the log:
"ACTUAL_TIME": "2023-09-03T08:06:51.946Z"

This is my logstash.conf filter:

filter{
  	date {
  		match => [ "ACTUAL_TIME", "yyyy-MM-dd HH:mm:ss.SSSZ" ]
  		remove_field => [ "ACTUAL_TIME" ]
	}
}

In discover, I see both fields, and they don’t match on the milliseconds:
image

Any help would be, like, really helpful! :slight_smile:

2

Hey @McJava1967

Maybe somethign like this

filter {
  mutate { remove_field => [ "field1", "field2", "field3", ... "fieldN" ] }
}

filter {

  	 mutate { remove_field => [ "ACTUAL_TIME"]  		
	}
   }

Is it posible to show your full Logstash config.

3

Thank you for responding!

This is the full (anonymized) logstash.conf. Note “contextMap” now prepends ACTUAL TIME. The “_app” above was to rename it. But still, it’s not working.

input {
   file {
     path => "/spring/logs/myApp/ELK/myLog.log"
   }
}
filter{
    json{
    	source => "message"
    }
  	date {
  		match => [ "contextMap.ACTUAL_TIME", "yyyy-MM-dd HH:mm:ss.SSSZ" ]
  		remove_field => [ "contextMap.ACTUAL_TIME" ]
	}
}output {
    opensearch {
        hosts => "http://myServer:9200"
        user => "XXXXX"
        password => "XXXXX"
        index => "my-app-%{[agent.version]}-%{+yyyy.MM.dd}"
        ssl_certificate_verification => false
    }
}
4

OK. This issue seems to be with inner JSON. I told it to just remove “host”, and that works.

We are sending the Log4j2 MDC map, which generates a section like this:

	"contextMap": {
		"ACTUAL_TIME": "2023-09-03T08:06:51.946Z",

Does anyone know of a way to remove just ACTUAL_TIME? I’ve tried all combinations I can think of.

Thanks!