# Unable to run 3 nodes simultaneously on same cluster **URL:** \ **Category:** Open Source Elasticsearch and Kibana\ **Created:** [August 4, 2020, 8:51am UTC](https://forum.opensearch.org/t/unable-to-run-3-nodes-simultaneously-on-same-cluster/3444 "2020-08-04T08:51:11Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![mvsn98](https://avatars.discourse-cdn.com/v4/letter/m/e19b73/32.png) [@mvsn98](https://forum.opensearch.org/u/mvsn98)\ **Post date:** [August 4, 2020, 8:51am UTC](https://forum.opensearch.org/t/unable-to-run-3-nodes-simultaneously-on-same-cluster/3444/1 "2020-08-04T08:51:11Z")
Hello, I am trying to configure a cluster with the following nodes: node-1 10.10.0.134 (Data Node) node-2 10.10.0.149 (Master Node) node-3 10.10.0.87 (Master and Data Node) The node.data , node.master and node.ingest parameters have been set up according to the above mentioned plan. I have set discovery.zen.minimum\_master\_nodes: 2 And also, cluster.initial\_master\_nodes: [“node-2”] discovery.seed\_hosts: [“10.10.0.134”,“10.10.0.149”,“10.10.0.87”] are present in the elasticsearch.yml file of all 3 nodes. However, when I start elasticsearch service on all 3 nodes, and try to use the following query: curl -XGET [https://10.10.0.134:9200/\_cat/nodes?v](https://10.10.0.134:9200/_cat/nodes?v) -u admin:admin --insecure somehow, I either get the result contains only those 2 nodes in the cluster on which I started the elasticsearch service (after editing yml file) before the 3rd node OR an “OpenDistro security is not initialized” message. The master is automatically elected out of the 2 nodes and the 3rd node is always left out. I am unable to include all 3 nodes in the same cluster at once. Can someone help me with this? ---
**Author:** ![sayaligaikawad](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/sayaligaikawad/32/4953_2.png) [@sayaligaikawad](https://forum.opensearch.org/u/sayaligaikawad)\ **Post date:** [August 4, 2020, 6:21pm UTC](https://forum.opensearch.org/t/unable-to-run-3-nodes-simultaneously-on-same-cluster/3444/2 "2020-08-04T18:21:04Z")
Hi @mvsn98, Can you share more information on the issue like what distribution are you using and the config file used? Thanks! ---
**Author:** ![mvsn98](https://avatars.discourse-cdn.com/v4/letter/m/e19b73/32.png) [@mvsn98](https://forum.opensearch.org/u/mvsn98)\ **Post date:** [August 5, 2020, 6:08am UTC](https://forum.opensearch.org/t/unable-to-run-3-nodes-simultaneously-on-same-cluster/3444/3 "2020-08-05T06:08:59Z")
@sayaligaikawad OpenDistro for Elasticsearch Version details are as follows: { “name” : “node-3”, “cluster\_name” : “test”, “cluster\_uuid” : “2ksXrXyWTsqRQHW5QeMHxg”, “version” : { “number” : “7.8.0”, “build\_flavor” : “oss”, “build\_type” : “deb”, “build\_hash” : “757314695644ea9a1dc2fecd26d1a43856725e65”, “build\_date” : “2020-06-14T19:35:50.234439Z”, “build\_snapshot” : false, “lucene\_version” : “8.5.1”, “minimum\_wire\_compatibility\_version” : “6.8.0”, “minimum\_index\_compatibility\_version” : “6.0.0-beta1” }, “tagline” : “You Know, for Search” } The contents of **Elasticsearch.yml** file are as follows: cluster.name: test # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: node-3 node.master: true node.data: true node.ingest: true # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /var/lib/elasticsearch # path.logs: /var/log/elasticsearch # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory\_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 10.10.0.149 # # Set a custom port for HTTP: # http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when this node is started: # The default list of hosts is [“127.0.0.1”, “[::1]”] # discovery.seed\_hosts: [“10.10.0.134”,“10.10.0.149”,“10.10.0.87”] discovery.zen.minimum\_master\_nodes: 2 # # Bootstrap the cluster using an initial set of master-eligible nodes: # cluster.initial\_master\_nodes: [“10.10.0.149”] # # For more information, consult the discovery and cluster formation module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover\_after\_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # #action.destructive\_requires\_name: true ######## Start OpenDistro for Elasticsearch Security Demo Configuration ######## # WARNING: revise all the lines below before you go into production opendistro\_security.ssl.transport.pemcert\_filepath: esnode.pem opendistro\_security.ssl.transport.pemkey\_filepath: esnode-key.pem opendistro\_security.ssl.transport.pemtrustedcas\_filepath: root-ca.pem opendistro\_security.ssl.transport.enforce\_hostname\_verification: false opendistro\_security.ssl.http.enabled: true opendistro\_security.ssl.http.pemcert\_filepath: esnode.pem opendistro\_security.ssl.http.pemkey\_filepath: esnode-key.pem opendistro\_security.ssl.http.pemtrustedcas\_filepath: root-ca.pem opendistro\_security.allow\_unsafe\_democertificates: true opendistro\_security.allow\_default\_init\_securityindex: true opendistro\_security.authcz.admin\_dn: - CN=kirk,OU=client,O=client,L=test, C=de opendistro\_security.audit.type: internal\_elasticsearch opendistro\_security.enable\_snapshot\_restore\_privilege: true opendistro\_security.check\_snapshot\_restore\_write\_privileges: true opendistro\_security.restapi.roles\_enabled: [“all\_access”, “security\_rest\_api\_access”] cluster.routing.allocation.disk.threshold\_enabled: false #node.max\_local\_storage\_nodes: 3 ######## End OpenDistro for Elasticsearch Security Demo Configuration ######## ---
**Author:** ![weicongs](https://avatars.discourse-cdn.com/v4/letter/w/48db29/32.png) [@weicongs](https://forum.opensearch.org/u/weicongs)\ **Post date:** [August 5, 2020, 6:26pm UTC](https://forum.opensearch.org/t/unable-to-run-3-nodes-simultaneously-on-same-cluster/3444/4 "2020-08-05T18:26:53Z")
@mvsn98 This configuration " discovery.zen.minimum\_master\_nodes: 2 " isn’t used by es. we can simply remove it. I don’t see any other issue with assumption that the configuration should be the same in all of these 3 nodes. could you help to share the logs of the node which isn’t added into the cluster? And ---
**Author:** ![mvsn98](https://avatars.discourse-cdn.com/v4/letter/m/e19b73/32.png) [@mvsn98](https://forum.opensearch.org/u/mvsn98)\ **Post date:** [August 6, 2020, 5:33am UTC](https://forum.opensearch.org/t/unable-to-run-3-nodes-simultaneously-on-same-cluster/3444/5 "2020-08-06T05:33:10Z")
Yes @weicongs Here are the logs of node-2 which does not get added to the cluster: [2020-08-05T10:22:37,286][WARN][o.e.c.c.ClusterFormationFailureHelper] [node-2] master not discovered or elected yet, an election requires a node with id [p8Qsf-VIQBmUeNhuHOYHeQ], have discovered [{node-2}{mtFALIClSa67PgZVRP\_uKw}{nH41RZsNQeKzuQj4L4TWYA}{10.10.0.87}{10.10.0.87:9300}{dimr}] which is not a quorum; discovery will continue using [10.10.0.134:9300, 10.10.0.149:9300] from hosts providers and [{node-1}{p8Qsf-VIQBmUeNhuHOYHeQ}{uIi77MSRRu2TOl3qwh7MYw}{10.10.0.134}{10.10.0.134:9300}{dimr}, {node-2}{mtFALIClSa67PgZVRP\_uKw}{nH41RZsNQeKzuQj4L4TWYA}{10.10.0.87}{10.10.0.87:9300}{dimr}] from last-known cluster state; node term 81, last-accepted version 428 in term 81 [2020-08-05T10:22:41,383][INFO][o.e.n.Node] [node-2] stopping … [2020-08-05T10:22:41,384][INFO][c.a.o.s.a.s.SinkProvider] [node-2] Closing InternalESSink [2020-08-05T10:22:41,384][INFO][c.a.o.s.a.s.SinkProvider] [node-2] Closing DebugSink [2020-08-05T10:22:41,601][INFO][o.e.n.Node] [node-2] stopped [2020-08-05T10:22:41,603][INFO][o.e.n.Node] [node-2] closing … [2020-08-05T10:22:41,623][INFO][o.e.n.Node] [node-2] closed ---
**Author:** ![weicongs](https://avatars.discourse-cdn.com/v4/letter/w/48db29/32.png) [@weicongs](https://forum.opensearch.org/u/weicongs)\ **Post date:** [August 6, 2020, 3:23pm UTC](https://forum.opensearch.org/t/unable-to-run-3-nodes-simultaneously-on-same-cluster/3444/6 "2020-08-06T15:23:52Z")
@mvsn98 you can check it again. if this node 2 has the same configurations: “cluster.name: test” and "cluster.initial\_master\_nodes: [“10.10.0.149”] ". And if they are the same and network is accessible between this node and other 2 nodes, that might be caused by the previous cluster state. you can try to reinstall each node to get a clean env. btw, 2 master nodes aren’t normal configuration. you can try to make all of these 3 nodes as master nodes, or just have 1 master node.