Logstash - Parsing the JSON message with two @timestmap in the original message

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS - 2.3.0

Describe the issue:
Hi, we have been migrating millions of logs from Data Dog to Opensearch and one of the service logs ( the message pasted below ) contains two @timestamps in original message, due to which its arising a conflict and throws a date parse time failure.

Original Log message :

message :{"**@timestamp**":"2023-07-12T10:12:46.842913+02:00","@version":"1","message":"{\"**@timestamp**\": \"2023-07-12 10:12:46\", \"event_user\": \"admin\", \"event\":\"finish\",\"state\": \"succeeded\", \"object\": \"prodovh_Environment\", \"user\": \"admin\", \"aborted_by\": \"-\",}","host":"test-prodovh.rod","severity":"info","selinux_enforced":"true","env":"prodovh","appenv":"common-prodovh","servertype":"mgmt"}

in logstash.conf

first I’m parsing the message with json module, this will split all the outer json fileds and value and results as below

“@timestamp”:“2023-07-12T10:12:46.842913+02:00”
“@version”:“1”
“message”:“{"@timestamp": "2023-07-12 10:12:46", "event_user": "admin", "event":"finish","state": "succeeded", "object": "prodovh_Environment", "user": "admin", "aborted_by": "-",}”
“host”:“test-prodvh.rod”
“severity”:“info”
“selinux_enforced”:“true”
“env”:“prodovh”
“appenv”:“prodvh”
“servertype”:“mgmt”

This will sync up correct timezone and seconds with opensearch

Further I am trying to parse the nested json “message” field it throws an date time parse failure because its trying to override the @timestamp field again.

I have tried renaming the @timestamp field using mutate filter and it doesn’t work and tried removing the field but It **omits the entire log message itself **

Can anyone give some ideas on how to deal with it please.

Configuration:

Relevant Logs or Screenshots:

2

How about this: use mutate with gsub (ugly, I know) to rename the inner @timestamp into something else. Especially if it’s always at the beginning, it should be safe. Then parsing should work :crossed_fingers:

1 Like
3

Can you please post/give an example with sample code

gsub with mutate

4

Hi Guys,

Can anyone please help me with this please.

5

This filter works well in my local env:

filter {
	json {
        source => "message"
      }
	mutate {
        rename => {"@timestamp" => "outer_timestamp"}
        gsub => ["message", "@timestamp", "inner_timestamp"]
     }
     json {
        source => "message"
      }
      date {
        match => [ "inner_timestamp", "yyyy-MM-dd HH:mm:ss" ]
      }
}

, by the way, the nested message field you put is not a valid json, and the inner @timestamp cannot be parsed successfully by json filter because of the format, so we have to rename it and use date filter to parse.

Here is the message I used to test:

{"@timestamp":"2023-07-12T10:12:46.842913+02:00","@version":"1","message":"{\"@timestamp\": \"2023-07-12 10:12:46\", \"event_user\": \"admin\", \"event\":\"finish\",\"state\": \"succeeded\", \"object\": \"prodovh_Environment\", \"user\": \"admin\", \"aborted_by\": \"-\"}","host":"test-prodovh.rod","severity":"info","selinux_enforced":"true","env":"prodovh","appenv":"common-prodovh","servertype":"mgmt"}
2 Likes
6

Hi @gaobinlong, sorry for the late response as I was on vacation and Thank you for the solution.
But I needed the outer timestamp to be @timestamp and nested @timestamp to be renamed / substituted so that It wont conflicts with @timestamp.

Finally, It worked
Thanks once again!

filter {
	json {
        source => "message"
      }
	mutate {
         gsub => ["message", "@timestamp", "inner_timestamp"]
     }
     json {
        source => "message"
      }
      date {
        match => [ "outer_timestamp", "yyyy-MM-dd HH:mm:ss.SSS" ]
      }
}
1 Like
7

Thanks @radhu for the initial idea, I was dumb enough to get this :smiley:

1 Like