3 nodes opensearch cluster, cannot start with only 2 nodes

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.4.0

Describe the issue:
I have 3 opensearch nodes, 1 master and 2 data (+master)
when I shutdown the two data nodes and start one of them back

it fails to start.

as far as I understand it should be able to start with only 2 nodes.

Configuration:
opensearch.yml

1
plugins.security.ssl.transport.pemcert_filepath: certificates/elk-transport-crt.pem
plugins.security.ssl.transport.pemkey_filepath: certificates/elk-transport-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certificates/elk-node-crt.pem
plugins.security.ssl.http.pemkey_filepath: certificates/elk-node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • ‘CN=CONTROL-M_EM_ES_admin,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM’

plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3
cluster.name: workflow_insights_cluster
network.host: 0
node.name: px-b24df8b7
node.roles: [“ingest”,“master”,“remote_cluster_client”]
http.port: 19200
transport.port: 19300
cluster.initial_master_nodes: [‘px-6cb6045b’]
discovery.seed_hosts: [‘px-b24df8b7’, ‘px-3e381f9f’, ‘px-6cb6045b’]
plugins.security.nodes_dn:

  • ‘CN=CONTROL-M_EM_ES_transport,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM’

path.logs: /home/pxone1/ctm_em/log/services/workflow_insights

2
opensearch.yml

plugins.security.ssl.transport.pemcert_filepath: certificates/elk-transport-crt.pem
plugins.security.ssl.transport.pemkey_filepath: certificates/elk-transport-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certificates/elk-node-crt.pem
plugins.security.ssl.http.pemkey_filepath: certificates/elk-node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • ‘CN=CONTROL-M_EM_ES_admin,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM’

plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3

path.logs: /home/pxem1/ctm_em/log/services/workflow_insights
cluster.name: workflow_insights_cluster
network.host: 0
node.name: px-3e381f9f
node.roles: [“data”,“ingest”,“master”,“remote_cluster_client”]
action.auto_create_index: false
bootstrap.memory_lock: true
http.port: 19200
transport.port: 19300
cluster.initial_master_nodes: [‘px-6cb6045b’]
discovery.seed_hosts: [‘px-3e381f9f’, ‘px-b24df8b7’, ‘px-6cb6045b’]
plugins.security.nodes_dn:

  • ‘CN=CONTROL-M_EM_ES_transport,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM’

3
opensearch.yml

plugins.security.ssl.transport.pemcert_filepath: certificates/elk-transport-crt.pem
plugins.security.ssl.transport.pemkey_filepath: certificates/elk-transport-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certificates/elk-node-crt.pem
plugins.security.ssl.http.pemkey_filepath: certificates/elk-node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: certificates/elastic_ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • ‘CN=CONTROL-M_EM_ES_admin,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM’

plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
node.max_local_storage_nodes: 3
path.logs: /home/pxem1/ctm_em/log/services/workflow_insights
cluster.name: workflow_insights_cluster
network.host: 0
node.name: px-6cb6045b
node.roles: [“initial_master”,“data”,“ingest”,“master”,“remote_cluster_client”]
action.auto_create_index: false
bootstrap.memory_lock: true
http.port: 19200
transport.port: 19300
cluster.initial_master_nodes: [‘px-6cb6045b’]
discovery.seed_hosts: [‘px-6cb6045b’, ‘px-3e381f9f’, ‘px-b24df8b7’]
plugins.security.nodes_dn:

  • ‘CN=CONTROL-M_EM_ES_transport,O=ChangeMe,L=ChangeMeL,ST=ChangeMeST,C=CM’

Relevant Logs or Screenshots:
opensearch log:

image
image2523×851 401 KB

when I try to run securityadmin:

image
image1175×1037 83.2 KB

2

@nap608 What was the status of the cluster when you stopped the node?
Were you able to run API calls against the remaining nodes?

3

@pablo

The status was green, yes I was able to.
If i set all nodes to up, the status would be green again.
I’m just trying to verify I have HA working here.

4

@nap608 Did you check the status of the indices and shards when the cluster was RED?

5

Hi @pablo
I think there is some confusion here.

I didn’t stop single node, I had 3 nodes, I stopped 2 of them, and then started 1 of the 2 I’ve stopped.

before I’ve stopped status was green.

but after I’ve stopped the 2 of them, opensearch did not come up anymore,
so how i’m suppose to check indices status in that scenario.

Is my scenario even valid?
does opensearch support cluster starting with only 2 nodes ?
or HA is only in case 1 of the nodes is stopping, and thats it.

6

Hi there,
if you want to achieve HA, you cannot take half or more nodes down.
From opensearch docs:
Three dedicated cluster manager nodes in three different zones is the right approach for almost all production use cases. This configuration ensures your cluster never loses quorum. Two nodes will be idle for most of the time except when one node goes down or needs some maintenance.
Creating a cluster - OpenSearch Documentation

If you have 3 nodes, you’re able to take only 1 node down at one time. In your case you took down 2 nodes of 3 so that’s why cluster didn’t behave properly.

I will also refer you to elasticsearch docs about quorum-based decision making (probably master election is not different in the opensearch) Quorum-based decision making | Elasticsearch Guide [8.12] | Elastic

7

Hi @Manual9960 ,
Thanks for your reply,
I’ve a follow up question:

In that scenario that two out of 3 nodes going down.

Is there some possibility of making the single node work? even in yellow status.
or restore one nodes and make it work with two nodes (that one was down)?
or the only solution is to restore all 3 nodes?

8

It’s impossible to achieve HA with only 1 node working

If you restore one node and in overall you have two nodes online (1 node offline) opensearch should work. The issue is that you have wrong configuration. Why do you have only one initial master node? You should list all of your master node, not only one.

Bootstrapping a cluster | Elasticsearch Guide [8.12] | Elastic

Also if you want achieve HA, please check if you have replica shard on your indicates
Optimize OpenSearch index shard sizes · OpenSearch

9

You touched a few different points

  1. Is a single node cluster supported?
    yes if you mean a single node from the very beginning. It is a special case. Configure discovery.type=single-node. See the docs: Docker - OpenSearch Documentation.

  2. Can it work with two (2) nodes?
    yes if you mean from the very beginning. However, it is a bad idea. You can have ‘split brain’ syndrome. Again the 2 nodes are not a result cluster failure.

I always run at least 3 nodes in a cluster. It just works better. I have done the single node and 2 but it was a flaky.

To your situation, I believe a cluster has to have at a minimum of a cluster_manager and a data node to run. So if both data nodes are down you are in a bad situation. In a cluster that has some nodes down but maintains a cluster_manager and data node it will be functional. The health status will be yellow or maybe red - not sure. When the other data nodes comes up it will join the cluster. Since you lost all data nodes the cluster state is kind of unknown. I would restart all nodes and cross my fingers.

10

Hi @Manual9960 ,

“It’s impossible to achieve HA with only 1 node working” =>
I’m not trying to achieve HA with 1 node, i’m trying to understand what happens in HA if 2 nodes out of 3 going down, is 1 or 2 are enough to temporiarily work?

“If you restore one node and in overall you have two nodes online (1 node offline) opensearch should work. The issue is that you have wrong configuration. Why do you have only one initial master node? You should list all of your master node, not only one.” =>
I’ve tried your suggestion and change ‘cluster.initial_master_nodes: [‘px-6cb6045b’]’
to include all nodes in cluster, and result is the same.
if I’ve set two nodes down and restore the ‘original’ master, the one I’ve ran security admin at, all is ok.
if I restore any other node apart from the originial master, cluster is down with this message:
(even though two nodes are up and running)

image
image1829×286 41.2 KB

any suggestions?

Hi @ssablan ,
look at this reply;
“It’s impossible to achieve HA with only 1 node working” =>
I’m not trying to achieve HA with 1 node, i’m trying to understand what happens in HA if 2 nodes out of 3 going down, is 1 or 2 are enough to temporiarily work?

Can it work with two (2) nodes? =>
not from begining only temporarily for HA purposes
my cluster is set with 3, i’m trying to make the HA work as it should.

any suggestions?

11

look up ^

12

@ssablan @Manual9960 @kris

?

13

I can say from experience that a 3-node cluster can operate with 2 nodes. I have a production cluster that when upgrading operates fine with 2 nodes while I am updating the OpenSearch version on the 3rd. I have never tried to run with 1 node of a 3-node cluster.
For a cluster to be operational it needs to have a voting quorum and at least one node performing each role. Since you mentioned there was only a master node the cluster is in an error state. You lost the quorum. Bringing the other nodes back up may or may not clear that cluster error. It is kind of an unknown/cluster failure state (to me anyway). To recover, I would shut down all nodes and then restart. That should force Discovery phase on all 3 nodes and the cluster will reform. A few other points.
• Changing the cluster.initial.master.node  this is used for bootstrapping the cluster initially. Since the cluster was already working, it should not be in the config.
• The pace at which the cluster-manager nodes drop from the cluster matters. The cluster has to update its settings for the voting/quorum. If the updates can’t occur, I think you end up again in a funky state.
• If you need to maintain operations with a loss of 2 nodes you are probably better of with a 5-node cluster.
There isn’t much more I can add. Best of luck.

14

I’m sorry I have to insist here, but you guys don’t seem to read or understand what I’ve said.
I’m not using only 1 node, I have 3 nodes, I’m shutting 2 nodes down, then starting 1 of them back up, so eventually I have 2 nodes.

and according to @Manual9960 , this scenario is legit and should work, but for me it doesn’t.
I’ve sent the error i’m getting.

So I would like to get a clear answer please.
Could I start 3 nodes cluster, with only 2 nodes (temporiarily) ?
If the answer is yes, then why i’m getting securityadmin error ?

Please read my question before answering.

15

Hi,
follow up questions. To summarize you have 3 node cluster node_1, node_2, node_3. Node_1 is “original master”. In your scenario you’re taking 2 nodes down - node_1 (original master) and node_2. If you bring only node_2 back, cluster is still down but if you bring back node_1 (instead of node_2), cluster is working with 2 nodes. Cluster can work with 2 nodes but one of them needs to be original master node.
Is that correct?
How do you check if cluster is down/up? Using http api (example curl on nodes that are still up) or opensearch dashboard?

16

To summarize you have 3 node cluster node_1, node_2, node_3. Node_1 is “original master”. In your scenario you’re taking 2 nodes down - node_1 (original master) and node_2. If you bring only node_2 back, cluster is still down but if you bring back node_1 (instead of node_2), cluster is working with 2 nodes. Cluster can work with 2 nodes but one of them needs to be original master node.

You are correct :slight_smile:

How do you check if cluster is down/up? Using http api (example curl on nodes that are still up) or opensearch dashboard?

all 3, I see dashboard down, I run ‘_cluster/health’ and see red, and I see the errors on log that I’ve sent earlier.

So is this the “expected” behavior? or does the cluster should work if I start node_2 ?

17

Changing the cluster.initial.master.node  this is used for bootstrapping the cluster initially. Since the cluster was already working, it should not be in the config.

I’ve tried that suggestion and as @ssablan correctly explains it didn’t change anything in my situation.

The pace at which the cluster-manager nodes drop from the cluster matters. The cluster has to update its settings for the voting/quorum. If the updates can’t occur, I think you end up again in a funky state.

Could you elaborate on this? assuming node dropped without updating quorum.
the funky state means can’t work at all? or is there any workaround?
could I force quorum before getting dropped?

18

@Manual9960 ?

19

Hey @nap608

If you trying to verify a HA then from the documentation you can lose one NODE, It’s advisable as @Manual9960 suggested to configure one of the DATA NODES also .

You can make them cluster-manager-eligible data nodes that will also be used for ingesting data:

node.roles: [ data, ingest ]

This is found here.

20

Please read first then reply @Gsmitt .

I thought when @Manual9960 finally understood my question he will give me an answer.
but hes no where to be found.