From seconds ago to date - index

OpenDistro
1

Hello,
I was wondering if it was possible to receive data containing values like “2 seconds ago” and, before entering them, transform them into “current date - 2 seconds ago”. Is it possible to make this calculation? Thank you

2

Theoretically it’s possible via an ingester processor in two way:

  1. you implement an ingester processor that to all the job in Java/Scala/kotlin. You take the string and generate the time object (this is most flexible because you can create unit tests and manage all the cases)

  2. using a painless ingester processor the pseudo code should be the following:

  • regex to extract number “(\d+) seconds ago”
  • string to int => delta
  • value=Offsetdatetime.now().minus(delta, TimeUnit.seconds).toString().replace(" ", “T”)
  • set the value to a field

To speed up development use the simulate.

BUT the main issue it’s that the time that you send the event and server one can be different, you have latency (delay) in trasmission, networking is “bastard” by default…
To mitigate these issues I suggest you to save the timestamp when you acquire event in the document and replace the OffsetDatetime with your event time.

Best regards,
Alberto

1 Like
3

I am happy to hear that it can be done.
some doubts: where and how can I create and save scripts in elastic? is there a manual? is there an editable folder with files inside?
seen like this is it also possible to save a configuration file on elastic, generated by a form in kibana?

4

@accat You need to define a pipeline ingestion.
Look to this post that describes the feature and the official documentation.
Or read a good book (Ingester and scripting chapters). :smile: I’m the author. :star_struck:

It’s quite easy, but you should learn a bit of Elasticsearch before doing things.
Ingester pipeline is one of the more powerful feature of Elasticsearch, but people hardly uses it.

2 Likes