Unable to run 3 nodes simultaneously on same cluster

Hello,

I am trying to configure a cluster with the following nodes:

node-1 10.10.0.134 (Data Node)
node-2 10.10.0.149 (Master Node)
node-3 10.10.0.87 (Master and Data Node)

The node.data , node.master and node.ingest parameters have been set up according to the above mentioned plan.

I have set discovery.zen.minimum_master_nodes: 2

And also,

cluster.initial_master_nodes: [“node-2”]

discovery.seed_hosts: [“10.10.0.134”,“10.10.0.149”,“10.10.0.87”]

are present in the elasticsearch.yml file of all 3 nodes.

However, when I start elasticsearch service on all 3 nodes, and try to use the following query:

curl -XGET https://10.10.0.134:9200/_cat/nodes?v -u admin:admin --insecure

somehow, I either get the result contains only those 2 nodes in the cluster on which I started the elasticsearch service (after editing yml file) before the 3rd node OR an “OpenDistro security is not initialized” message.

The master is automatically elected out of the 2 nodes and the 3rd node is always left out.

I am unable to include all 3 nodes in the same cluster at once. Can someone help me with this?

Hi @mvsn98,

Can you share more information on the issue like what distribution are you using and the config file used?

Thanks!

@sayaligaikawad

OpenDistro for Elasticsearch Version details are as follows:

{
“name” : “node-3”,
“cluster_name” : “test”,
“cluster_uuid” : “2ksXrXyWTsqRQHW5QeMHxg”,
“version” : {
“number” : “7.8.0”,
“build_flavor” : “oss”,
“build_type” : “deb”,
“build_hash” : “757314695644ea9a1dc2fecd26d1a43856725e65”,
“build_date” : “2020-06-14T19:35:50.234439Z”,
“build_snapshot” : false,
“lucene_version” : “8.5.1”,
“minimum_wire_compatibility_version” : “6.8.0”,
“minimum_index_compatibility_version” : “6.0.0-beta1”
},
“tagline” : “You Know, for Search”
}

The contents of Elasticsearch.yml file are as follows:

cluster.name: test

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: node-3

node.master: true
node.data: true node.ingest: true

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 10.10.0.149

Set a custom port for HTTP:

http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.seed_hosts: [“10.10.0.134”,“10.10.0.149”,“10.10.0.87”]
discovery.zen.minimum_master_nodes: 2

Bootstrap the cluster using an initial set of master-eligible nodes:

cluster.initial_master_nodes: [“10.10.0.149”]

For more information, consult the discovery and cluster formation module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

#action.destructive_requires_name: true

######## Start OpenDistro for Elasticsearch Security Demo Configuration ########

WARNING: revise all the lines below before you go into production

opendistro_security.ssl.transport.pemcert_filepath: esnode.pem
opendistro_security.ssl.transport.pemkey_filepath: esnode-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: esnode.pem
opendistro_security.ssl.http.pemkey_filepath: esnode-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: root-ca.pem
opendistro_security.allow_unsafe_democertificates: true
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de

opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
cluster.routing.allocation.disk.threshold_enabled: false
#node.max_local_storage_nodes: 3
######## End OpenDistro for Elasticsearch Security Demo Configuration ########

@mvsn98 This configuration " discovery.zen.minimum_master_nodes: 2 " isn’t used by es. we can simply remove it. I don’t see any other issue with assumption that the configuration should be the same in all of these 3 nodes. could you help to share the logs of the node which isn’t added into the cluster? And

Yes @weicongs

Here are the logs of node-2 which does not get added to the cluster:

[2020-08-05T10:22:37,286][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-2] master not discovered or elected yet, an election requires a node with id [p8Qsf-VIQBmUeNhuHOYHeQ], have discovered [{node-2}{mtFALIClSa67PgZVRP_uKw}{nH41RZsNQeKzuQj4L4TWYA}{10.10.0.87}{10.10.0.87:9300}{dimr}] which is not a quorum; discovery will continue using [10.10.0.134:9300, 10.10.0.149:9300] from hosts providers and [{node-1}{p8Qsf-VIQBmUeNhuHOYHeQ}{uIi77MSRRu2TOl3qwh7MYw}{10.10.0.134}{10.10.0.134:9300}{dimr}, {node-2}{mtFALIClSa67PgZVRP_uKw}{nH41RZsNQeKzuQj4L4TWYA}{10.10.0.87}{10.10.0.87:9300}{dimr}] from last-known cluster state; node term 81, last-accepted version 428 in term 81
[2020-08-05T10:22:41,383][INFO ][o.e.n.Node ] [node-2] stopping …
[2020-08-05T10:22:41,384][INFO ][c.a.o.s.a.s.SinkProvider ] [node-2] Closing InternalESSink
[2020-08-05T10:22:41,384][INFO ][c.a.o.s.a.s.SinkProvider ] [node-2] Closing DebugSink
[2020-08-05T10:22:41,601][INFO ][o.e.n.Node ] [node-2] stopped
[2020-08-05T10:22:41,603][INFO ][o.e.n.Node ] [node-2] closing …
[2020-08-05T10:22:41,623][INFO ][o.e.n.Node ] [node-2] closed

@mvsn98 you can check it again. if this node 2 has the same configurations: “cluster.name: test” and "cluster.initial_master_nodes: [“10.10.0.149”] ". And if they are the same and network is accessible between this node and other 2 nodes, that might be caused by the previous cluster state. you can try to reinstall each node to get a clean env. btw, 2 master nodes aren’t normal configuration. you can try to make all of these 3 nodes as master nodes, or just have 1 master node.