Hi team,
We have a cluster hosted on AWS EC2 instances having
1 kibana (8GB root volume, non persistent)
3 master (8GB root volume, non-persistent)
12 data nodes (200GB persistent disk is attached per node)
We are using EC2 discovery plugin for discovering the cluster nodes. Due to some reasons, our master and kibana autoscaling groups recreated new instances and the older instances got replaced.
Current situation:
Master and Kibana nodes have new cluster UUIDS and the data nodes are running with older cluster UUID. Due to this, we are getting the following errors
2022-06-20T12:36:58,952][WARN ][o.e.c.c.Coordinator ] [127.0.0.1-node1] failed to validate incoming join request from node [{127.0.0.1-node2}{ADsjli4JQzKtHRdmdp-bDA}{ZoGZPrbFScmb2xVutbI3cA}{10.193.20.56}{10.193.20.56:9300}{dilrt}{aws_availability_zone=eu-west-1a, ml.machine_memory=33252052992, ml.max_open_jobs=20, node_type=hot, xpack.installed=true, transform.node=true}]
org.elasticsearch.transport.RemoteTransportException: [127.0.0.1-node2][10.193.20.56:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid vJoCmAqBRmS_zw5Q1g02iQ than local cluster uuid lY1hx_21SVmxrdXAmuYElA, rejecting
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:153) ~[elasticsearch-7.8.0.jar:7.8.0]
at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceivedDecorate(OpenDistroSecuritySSLRequestHandler.java:182) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityRequestHandler.messageReceivedDecorate(OpenDistroSecurityRequestHandler.java:285) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceived(OpenDistroSecuritySSLRequestHandler.java:142) ~[?:?]
at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin$7$1.messageReceived(OpenDistroSecurityPlugin.java:613) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:263) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) ~[elasticsearch-7.8.0.jar:7.8.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.0.jar:7.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
How can we make these masters join the old cluster? Any support would be highly appreciated.
We are also getting opendistro security plugin not initialized on master and Kibana nodes. I believe this is because of the above specified issue.