Problem with mapping data (timestamp) types by using Bulk API

OpenSearch
1

Hello.
I’ve created a mapping like this (file is called “plantilla.json”)…

{
   "index_patterns": ["packets-*"],
   "mappings": {
     "properties": {
         "timestamp": {"type":"date", "format":"epoch_millis" }
     }
   }
}

…by running this command: curl -X PUT -k -u “admin:admin” https://127.0.0.1:9200/_template/plantilla1 -H “Content-Type:application/json” -d @plantilla.json

I want to index a content like this (file is called “captura.json”…it’s really obtained from tshark -T ek 's output but here I’m showing a simplified version)…

{"index":{"_index":"packets-2022-04-08","_type":"doc"}}
{"timestamp":"1649452291404","layers":"1234"}
{"index":{"_index":"packets-2022-04-08","_type":"doc"}}
{"timestamp":"1649452291404","layers":"5678"}

…by running this command: curl -XPOST -k -u “admin:admin” https://127.0.0.1:9200/_bulk -H “Content-Type:application/json” --data-binary @captura.json

But I get this error:

{“took”:545,“errors”:true,“items”:[{“index”:{“_index”:“packets-2022-04-08”,“_type”:“doc”,“_id”:“7TwCE4ABMXOQX3ZKhwV-”,“status”:400,“error”:{“type”:“illegal_argument_exception”,“reason”:“mapper [timestamp] cannot be changed from type [date] to [text]"}}},{“index”:{”_index":“packets-2022-04-08”,“_type”:“doc”,“_id”:“7jwCE4ABMXOQX3ZKhwV-”,“status”:400,“error”:{“type”:“illegal_argument_exception”,“reason”:"mapper [timestamp] cannot be changed from type [date] to [text]"}}}]}

The shocking issue is that if I try to use the Index API (via PUT or POST method, it’s the same) to index individuals documents, with the same format in “timestamp” field, mapping just works. So there must be something wrong with Bulk API…

Thanks a lot!!

2

Try removing the quotes from your value.

“timestamp”:1649452291404

3

Thanks, but this just change the error message to: “mapper [timestamp] cannot be changed from type [date] to [long]”

Anyway, as I’ve said, these json lines are meant to be automatically created by tshark’s output. This means timestamp value it’s automatically written between quotation marks, it’s not my choice.

Thanks!

4

The other part that seems suspicious is: “_type”:“doc”

I’m not familiar with tshark, but I suspect it was designed before elasticsearch 6.x, and is attempting to use multiple document types. The only valid value for _type in 7.x and up is “_doc”. Not “doc”.

I think your issue is ultimately a tshark problem, and its compatibility situation with elasticsearch 7.x and up. (Opensearch, of course, is subject to this same breaking change of not supporting multiple document types.)

I googled this for you, and found multiple workarounds. Here’s one: ZENETYS - Tips & Tricks : PCAP to Elastic... make it work!

1 Like
5

YES, YOU’RE RIGHT!! It was that!!
Oooh, thanks a lot!!! THANKS!!!
I’ll write an issue to Wireshark’s github right now.

P.S: Here it is: https://gitlab.com/wireshark/wireshark/-/issues/18036

1 Like
6

Sorry, I don’t know how to mark this post as “Solved” or reference above specific message as the right solution…

1 Like