OS cluster manual snapshot cause search thread pool rejection spike

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): AWS OS managed cluster 2.19

Describe the issue:

Configure to schedule event bridge scheduler every 1 hour to trigger manual snapshot using snapshot api _snapshot to s3 bucket.

And found out each 1 hour, especially 8AM, when search thread pool rejection spike up. Im suspecting manual snapshot trigger this issue because the same exact time happen for nearly every day.

Current setup with 3 master node m7g.large.search and 10 data nodes r7g.4xlarge. How should I improve this? or any advice i should look for to find out the problem

Configuration:

Relevant Logs or Screenshots:

2

Hi @kitkit

How much is the data size ?

If you are taking incremental backup every hour , how much data is being written in that last one hour?

Is the spike noticed for each and every hour or specifically at 8 am everyday?

Also check the master node logs for details about any hot threads/issues/additional details if any?

3

@kitkit Are the S3 bucket and OpenSearch cluster in the same region?