We are using AWS Opensearch 1.0 as a document database to serve our retail users. We use Apache Kafka to sync Opensearch. This use-case requires Opensearch documents to be refreshed near real time for searching. We have observed that changes do not reflect for a long time.
Here is the summary of how we write and read in Openearch.
- Kafka event is received
- Parse object in the client application
- Client application matches received objects id to a matching Id in opensearch through the GET API.
- If documnt exists, update call is made, PUT.
- Customers are searching for data using Opensearch search API and changes are often not reflected for them.
- No errors or exceptions have been observed in Opensearch cluster.
Note: we have a lot of writes so a lot of indexing going on simultaneously to searches.
I suspect this is some refresh configuration that needs to be updated.