Opensearch heavy write problem

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): AWS Opensearch managed cluster version 2.8

Describe the issue:

Setup: Monthly index and I need to update specific candidate documents that exist in 36 months.

The update could be 150k per minute to use bulk update all documents and the problem is the write seems heavy and could impact search performance.

1 candidate could consists 2k documents to update. Any options or tools or solution to able to update heavily without causing performance issue.

Configuration:

Relevant Logs or Screenshots:

@kitkit Each bulk update re-reads and re-indexes the whole document, so 150k updates/min is heavy on the data nodes that also serve your searches. You could consider the following on your 2.8:

  • Raise refresh_interval on the indices being updated while the updates run.
  • Send updates in smaller, steady batches, preferably off-peak.
  • Add more or larger data nodes, and faster storage (higher IOPS), if the cluster is still saturated.

You could also consider using search replicas; however, these are available in 3.x.

Search replicas isolate search from the update load. Dedicated search nodes serve the queries while the primaries take the updates. This needs remote-backed storage and segment replication: Separate index and search workloads | OpenSearch Documentation

@pablo

  • Raise refresh_interval on the indices being updated while the updates run.
    I cant because impacting the update real time in front end
  • Send updates in smaller, steady batches, preferably off-peak.
    • Currently bulk 10 document to publish to opensearch. any other smaller method?
  • 4xlarge nodes now and already significant cost to support
  • Search replicas not supported in AWS opensearch managed cluster. let me know I can apply this in AWS env

one thing that jumps out is the bulk size of 10, thats very small for 150k a minute and the per request overhead will eat you alive. id try the opposite and tune by size instead, something like 5 to 10 mb per bulk with only a couple of bulk requests in flight at once, then watch for rejections and merge pressure. it wont make the updates themselves cheap since each one still reindexes the doc, but it should be a lot less wasteful than thousands of tiny requests. if the same candidate fields keep changing, moving those fields out into a separate small index is the bigger win long term.