Opensearch Cluster Highavailability testing question

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.4.0

Describe the issue:
I have 3 nodes cluster:
node_1, node_2, node_3.
Node_1 is “original master”.
In my scenario i’m taking 2 nodes down - node_1 (original master) and node_2.
If I bring only node_2 back, cluster is still down
but if I bring back node_1 (instead of node_2), cluster is working with 2 nodes.
Cluster can work with 2 nodes but one of them needs to be original master node.

Is this the expected behavior ?
or should the cluster work no matter which node I bring back?

Configuration:

opensearch.yml

plugins.security.ssl.transport.pemcert_filepath: certificates/elk-transport-crt.pem
plugins.security.ssl.transport.pemkey_filepath: certificates/elk-transport-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false 
plugins.security.ssl.http.enabled: true 
plugins.security.ssl.http.pemcert_filepath: certificates/elk-node-crt.pem
plugins.security.ssl.http.pemkey_filepath: certificates/elk-node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.allow_unsafe_democertificates: true 
plugins.security.allow_default_init_securityindex: true 
plugins.security.authcz.admin_dn: 
  - 'CN=CONTROL-M_EM_ES_admin,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM'

 
#plugins.security.audit.type: internal_opensearch 
plugins.security.enable_snapshot_restore_privilege: true 
plugins.security.check_snapshot_restore_write_privileges: true 
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"] 
plugins.security.system_indices.enabled: true 
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"] 
node.max_local_storage_nodes: 3 
######## End OpenSearch Security Demo Configuration ######## 
path.logs: /home/pxem1/ctm_em/log/services/workflow_insights
cluster.name: workflow_insights_cluster
network.host: 0
node.name: px-859b65ab
node.roles: ["initial_master","data","ingest","master","remote_cluster_client"]
action.auto_create_index: false
bootstrap.memory_lock: true
http.port: 19200
transport.port: 19300
cluster.initial_master_nodes: ['px-859b65ab']
discovery.seed_hosts: ['px-859b65ab']
plugins.security.nodes_dn: 
  - 'CN=CONTROL-M_EM_ES_transport,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM'

Relevant Logs or Screenshots:

the errors i’m receveing on node_2 when it doesn’t go up.

image
image1829×286 41.2 KB

Please provide an answer.

2

If you want that no matter which node goes down then the cluster can elect a new master, then you should add the 3 nodes to the settings cluster.initial_master_nodes and discovery.seed_hosts, and make sure each node has the same 2 settings.

3

@gaobinlong
I’ve already tried that, it did not solve the issue

I will quote @ssablan

Changing the cluster.initial.master.node  this is used for bootstrapping the cluster initially. Since the cluster was already working, it should not be in the config.