I have a 3-node OpenSearch cluster with the following roles:
Node 1: master, data (PROD)
Node 2: master, data (PROD)
Node 3: master, data (DR)
If two of the nodes go down, leaving only one node running, will the remaining node still:
Allow search queries?
Accept new indexing requests for existing indices?
Configuration: what i observed, if the 2 nodes are down and when i try to see the cluster status or nodes status in the browser (https://hostname:9200/_cat/nodes?v) getting Cluster_Discovered_Not_Found Exception and no URL & API calls are working.
But , In Elasticsearch is Quite different, even 2 nodes are down still 3rd node will work for search and existing indexing. (please correct me if iam wrong)
Additionally, is there any configuration or setting that influences this behavior (e.g., quorum requirements, discovery.type, cluster.routing.allocation.*)?
In OpenSearch also we can achieve 3rd node will work as like elastic search ?? otherwise i have to declare to my team to use 5 nodes (3 prod & 2 DR) as 2 nodes is required to form a quorum ??
Please advise and share the necessary links to me to explain to the team .
@vinay If 2 out of the 3 cluster-manager eligible nodes go down the new cluster-manager (master) will not be able to be elected.
Therefore if you are planning for a situation where 2 cluster-manager eligible might go down, you would need to create cluster with 5 cluster-manager eligible nodes, as you pointed out correctly yourself.
Regarding DR, have you explored the option of Snapshots and/or Cross-cluster replication as an alternative?
@Anthony Thank you so much for kind reply and apprciate it.
Before suggest the 5 Node Setup to my team , I need to know some further details, and just confirm that DR Cross Replication will not work for my case as because my entire arch is Active/Active.
why opensearch 3 nodes cluster, if 2 nodes are down and even with 1 node up and cannot work with existing Search ? But Elastic search also same ??? Heard from some of my team members, elastic search will work with one node for existing search even 2 nodes are down. is it correct?
If i suggest my team to use 5 node cluster (3 PROD & 2 DR) - can i use data roles for 5 nodes as (master,data) ? or can you suggest the best option to maintain the nodes roles.
After we created the cluster (3 node or 5 Node), we have to integrate with bitbucket. is there any thing we can follow as standard practice ?
Can you suggest the best Sharding concept for 3 Node & 5 Nodes clusters ? like how many Primary/Replicas can be put for 3 Node & 5 Node clusters??
@vinay It is possible to search cluster when there cluster_manager (master) is not determined, however the results are only sourced from available shards and are operating without coordination/cluster-state changes, therefore designing for one node left which will be serving reads is not a good idea. There is a setting to control this: cluster.no_cluster_manager_block, default is “write”, which blocks any write operation.
If you do decide to use this, You would need to ensure that at least one shard (primary or replica) is present on the remaining node. Therefore in a case of 3 nodes, you would need 1 primary and 2 replicas.
Regarding the DR, are you looking to stretch one cluster across 2 sites, am I understanding this correctly?
Sharding is a difficult one, because it really depends on your use case. As a general rule:
Search-heavy: ~10–30 GB per shard
Write/log-heavy: ~30–50 GB per shard
The number of primaries would usually be = index_size / target_shard_size