Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
1.2.4
Describe the issue:
Not able to setup cluster on AWS ec2 instances with use of discovery-ec2 plugin.
Am trying to setup of cluster with :
3 - master nodes
3 - data nodes
1- client
where each node created by the Autoscaling gruops.
ie: MasterAsg, DataAsg and ClientAsg
Based on the Asgs functionality, all nodes at a time, parallely getting created. Due to this we are not able to collec/capture the node IP/Hostname and at the same time not able to setup the config “cluster.initial_master_nodes” on opensearch.yml file…
So, Does the “cluster.initial_master_nodes” is mandatory to setup/form a brand new cluster?
is there any otherway to setup cluster?
In the below error log can see clearly… 7 nodes are discovered by plugin but not able to setup master node on cluster.
Configuration:
path:
logs: LOGPATH
data: DATAPATH
cluster.name: CLUSTER_NAME
node.name: NODE_NAME
node.roles: [ NODE_ROLE ]
#discovery-ec2 plugin configs
discovery.seed_providers: ec2
discovery.ec2.tag.cluster_nodes: CLUSTER_NAME
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
Relevant Logs or Screenshots:
[2022-12-01T14:22:12,800][WARN ][o.o.c.c.ClusterFormationFailureHelper] [i-0644829f120faaec5] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{i-0644829f120faaec5}{aPgha9Y2TZ2OpOOhephYgQ}{z4F-Uzi7RYSOPvHfWONwgA}{10.114.42.50}{10.114.42.50:9300}{m}{aws_availability_zone=us-east-1b, shard_indexing_pressure_enabled=true}, {i-0090fe2379eecdaf4}{MHz49zZ4T1yBSmzXBy-9QQ}{JEigClTFRR69tCLAVxmcKQ}{10.114.51.114}{10.114.51.114:9300}{m}{aws_availability_zone=us-east-1c, shard_indexing_pressure_enabled=true}]; discovery will continue using [127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305, 10.114.42.50:9300, 10.114.46.180:9300, 10.114.45.46:9300, 10.114.53.148:9300, 10.114.51.114:9300, 10.114.38.138:9300, 10.114.33.66:9300] from hosts providers and [{i-0644829f120faaec5}{aPgha9Y2TZ2OpOOhephYgQ}{z4F-Uzi7RYSOPvHfWONwgA}{10.114.42.50}{10.114.42.50:9300}{m}{aws_availability_zone=us-east-1b, shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-12-01T14:22:12,849][ERROR][o.o.s.c.ConfigurationRepository] [i-0644829f120faaec5] Cannot apply default config (this is maybe not an error!)
org.opensearch.discovery.MasterNotDiscoveredException: null
at org.opensearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:275) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:369) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:287) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:692) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) ~[opensearch-1.2.4.jar:1.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]