Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.11.1
OpenSearch operator 2.6.0
Describe the issue:
We have OpenSearch running in a Kubernetes cluster that is managed by the operator. We are using the OpenSearch cluster health API - Cluster health - OpenSearch Documentation to determine the cluster’s health. We consider green as healthy and yellow and red as unhealthy. We get alerts if it the cluster is not healthy.
The parameters we use are as follows
/_cluster/health?wait_for_status=green&timeout=15s
When we do a configuration change, the operator performs a rolling restart of the cluster. When a node is being restarted, the health check indicates that the cluster health is yellow. However this is expected when a restart is happening. But this triggers alerts which are false positive.
So my questions is, what can we do to improve the health check? Is there a parameter where we could say, if a rolling restart is going on, consider yellow as healthy. But at other times, consider only green as healthy?
Configuration:
3 dedicated master nodes
3 dedicated data nodes
Relevant Logs or Screenshots: