Dynamically filter and index data from existing index to new index in OpenSearch

OpenSearch
1

I am looking for guidance on dynamically filtering and indexing data from an existing index to a new index in OpenSearch (version 2.11.0). Specifically, I have an existing index containing logs from an e-commerce application, and I want to create a new index with only the filtered logs based on certain criteria, without any grouping or aggregation.

Here are the key requirements:

1.The new index should contain all fields present in the filtered logs.

2.Filtering should be dynamic, meaning that new logs meeting the filter criteria should be automatically indexed into the new index.

3.No aggregation or grouping is required,each document should be indexed individually.

I have explored options such as using Watcher, but it seems that Watcher functionality is not available in OpenSearch. Can someone suggest an alternative approach or tooling that can help achieve this requirement?

Additionally, I would appreciate any insights or best practices on how to handle updates and deletions from the source index and ensure synchronization with the new index.

Thank you for your assistance.

2

Why not doing these things in the client side? Write two indices at the same time, one for complete data and another for filtered data. In OpenSearch, reindex can help to query a index and write the results to another index, but it runs only one time and cannot handle updates and deletions.