New content read/write capabilities (mirroring)

Data Prepper
1

I need to capture new documents posted to OpenSearch, and re-post them to some security tooling. I’m curious if the Data-Prepper is the best way to go about doing so. The current requirement is just “from now, forward new messages”, and I can select from a great number of export formats.

Basically, I’d like to subscribe to a “stream” of all new events.

To me, it looks like a data prepper with an opensearch source and an opensearch destination is the way to go, but no documentation states where the “I’ve got up to here” registry resides, so I don’t know which volumes/files I have to preserve the state of. Additionally, it looks like I’d only be able to provide a “start here” marker by way of a query statement.

Am I on the best path, or is there a better way to accomplish my goals?

2

Hello @jeffk,
How do you access your documents in OpenSearch?
There are various ways to achieve your goal, depending on the situation.
For example, if you index your documents based on Logstash, you can configure an additional output to write the same document to another destination, e.g. OpenSearch, Elasticsearch, Redis or Kafka.
If you cannot control the import because an application is in front of OpenSearch, you can use Logstash with the OpenSearch Input Plugin and output the document to another OpenSearch.
If you are comfortable with replication, you can also use cross cluster

1 Like
3

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.