Opensearch 1.2.4 - cluster formation issue : master not discovered yet

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):


Describe the issue:

Not able to setup cluster on AWS ec2 instances with use of discovery-ec2 plugin.

Am trying to setup of cluster with :
3 - master nodes
3 - data nodes
1- client

where each node created by the Autoscaling gruops.
ie: MasterAsg, DataAsg and ClientAsg

Based on the Asgs functionality, all nodes at a time, parallely getting created. Due to this we are not able to collec/capture the node IP/Hostname and at the same time not able to setup the config “cluster.initial_master_nodes” on opensearch.yml file…

So, Does the “cluster.initial_master_nodes” is mandatory to setup/form a brand new cluster?

is there any otherway to setup cluster?

In the below error log can see clearly… 7 nodes are discovered by plugin but not able to setup master node on cluster.


node.roles: [ NODE_ROLE ]

#discovery-ec2 plugin configs
discovery.seed_providers: ec2
discovery.ec2.tag.cluster_nodes: CLUSTER_NAME
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone

Relevant Logs or Screenshots:

[2022-12-01T14:22:12,800][WARN ][o.o.c.c.ClusterFormationFailureHelper] [i-0644829f120faaec5] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{i-0644829f120faaec5}{aPgha9Y2TZ2OpOOhephYgQ}{z4F-Uzi7RYSOPvHfWONwgA}{}{}{m}{aws_availability_zone=us-east-1b, shard_indexing_pressure_enabled=true}, {i-0090fe2379eecdaf4}{MHz49zZ4T1yBSmzXBy-9QQ}{JEigClTFRR69tCLAVxmcKQ}{}{}{m}{aws_availability_zone=us-east-1c, shard_indexing_pressure_enabled=true}]; discovery will continue using [,,,,, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305,,,,,,,] from hosts providers and [{i-0644829f120faaec5}{aPgha9Y2TZ2OpOOhephYgQ}{z4F-Uzi7RYSOPvHfWONwgA}{}{}{m}{aws_availability_zone=us-east-1b, shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-12-01T14:22:12,849][ERROR][o.o.s.c.ConfigurationRepository] [i-0644829f120faaec5] Cannot apply default config (this is maybe not an error!)
org.opensearch.discovery.MasterNotDiscoveredException: null
at$AsyncSingleAction$2.onTimeout( ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout( ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout( ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cluster.service.ClusterApplierService$ ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.common.util.concurrent.ThreadContext$ ~[opensearch-1.2.4.jar:1.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker( ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$ ~[?:?]
at [?:?]

Hey @kmaheshgupta786

If this is Docker/Docker-compose what does the configuration look like (i.e., docker-compose.yml)
Normally when you see this in the logs

Either its taking time to fire up or the network configuration/s are not correct, firewall or something is blocking the port/s neeeded to be discovered. But that is a Warning, i see your error ,

Perhaps the network connection/s need to be resolved. I havent used AWS ec2 but I have had this isse before.