Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch version 2.11.1
Describe the issue:
I have a service that do query from opensearch on the regular.
For some reason, that service has been getting timeout errors around 0:00, 09:00, and 18:00 UTC
Especially, Timeouts are a definite occurrence at 18:00.
When monitoring the ‘Document Indexed Rate’ on Grafana,
I can see the rates crossing over for different data nodes.
Interestingly, the times when these crossovers occur are very similar to the times when we experience timeouts.
How can we resolve this timeout issue?
Configuration:
There are 3 master node, 9 data node in the OpenSearch Cluster
master node’s role is only master
and data node’s role is " NODE_ROLES=data,ingest,remote_cluster_client,dynamic"
Relevant Logs or Screenshots:
