Opensearch Cluster Highavailability testing question

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:
I have 3 nodes cluster:
node_1, node_2, node_3.
Node_1 is “original master”.
In my scenario i’m taking 2 nodes down - node_1 (original master) and node_2.
If I bring only node_2 back, cluster is still down
but if I bring back node_1 (instead of node_2), cluster is working with 2 nodes.
Cluster can work with 2 nodes but one of them needs to be original master node.

Is this the expected behavior ?
or should the cluster work no matter which node I bring back?


opensearch.yml certificates/elk-transport-crt.pem certificates/elk-transport-key.pem certificates/elastic_ca.pem false true certificates/elk-node-crt.pem certificates/elk-node-key.pem certificates/elastic_ca.pem true true 
  - 'CN=CONTROL-M_EM_ES_admin,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM' internal_opensearch true true ["all_access", "security_rest_api_access"] true [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"] 
node.max_local_storage_nodes: 3 
######## End OpenSearch Security Demo Configuration ######## 
path.logs: /home/pxem1/ctm_em/log/services/workflow_insights workflow_insights_cluster 0 px-859b65ab
node.roles: ["initial_master","data","ingest","master","remote_cluster_client"]
action.auto_create_index: false
bootstrap.memory_lock: true
http.port: 19200
transport.port: 19300
cluster.initial_master_nodes: ['px-859b65ab']
discovery.seed_hosts: ['px-859b65ab'] 
  - 'CN=CONTROL-M_EM_ES_transport,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM'

Relevant Logs or Screenshots:

the errors i’m receveing on node_2 when it doesn’t go up.

Please provide an answer.

If you want that no matter which node goes down then the cluster can elect a new master, then you should add the 3 nodes to the settings cluster.initial_master_nodes and discovery.seed_hosts, and make sure each node has the same 2 settings.

I’ve already tried that, it did not solve the issue

I will quote @ssablan

Changing the cluster.initial.master.node  this is used for bootstrapping the cluster initially. Since the cluster was already working, it should not be in the config.