Data prepper not Parsing Epoch Millis timestamp with decimals correctly

Data Prepper
, , ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): Latest version

Describe the issue:
Data prepper is not parsing the epoch millis time stamp with decimals correctly. Float type (index template format) is however able to parse it correctly.
Log Error message is self explanatory

Relevant Logs or Screenshots:

java.lang.RuntimeException: failed to parse field [date] of type [date] in document with id 'xaXpVYcBu2BnbvCiOhMJ'. 
Preview of field's value: '1.680773560695857E9' caused by failed to parse date field 
[1.680773560695857E9] with format [epoch_millis] caused by Failed to parse 
with all enclosed parsers

If anyone knows any workaround to fix it, please share

2

@hardikbajaj I believe it is Open Search that wasn’t able to parse the field. Looking at this doc, epoch_millis parses a long value that corresponds to milliseconds since the epoch. But the field given (1.680773560695857E9) seems to be in microseconds, which is not supported.

A workaround would be to pre-process and convert the field so it’s in milliseconds before sending to OpenSearch. I’m not sure there’s a Data Prepper processor that can do that currently. But feel free to open an issue in Issues · opensearch-project/data-prepper · GitHub with the problem you are facing or any request you have.

3

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.