# Nodes fall out of the cluster es 7.9.1 **URL:** \ **Category:** Open Source Elasticsearch and Kibana\ **Created:** [November 23, 2020, 3:53pm UTC](https://forum.opensearch.org/t/nodes-fall-out-of-the-cluster-es-7-9-1/4259 "2020-11-23T15:53:17Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![ThreatInter](https://avatars.discourse-cdn.com/v4/letter/t/49beb7/32.png) [@ThreatInter](https://forum.opensearch.org/u/ThreatInter)\ **Post date:** [November 23, 2020, 3:53pm UTC](https://forum.opensearch.org/t/nodes-fall-out-of-the-cluster-es-7-9-1/4259/1 "2020-11-23T15:53:17Z")
Hello, we have the following problem: nodes can periodically fall out of the cluster ```auto [2020-11-19T09:00:37,415][INFO][o.e.c.s.MasterService] [h1-es03] node-left[{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr} reason: followers check retry count exceeded], term: 87, version: 236727, delta: removed {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr}} [2020-11-19T09:00:39,763][INFO][o.e.c.s.ClusterApplierService] [h1-es03] removed {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr}}, term: 87, version: 236727, reason: Publication{term =87, version=236727} [2020-11-19T09:00:47,890][INFO][o.e.c.s.MasterService] [h1-es03] node-join[{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr} join existing leader], term: 87, version: 236730, delta: added {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr}} [2020-11-19T09:00:52,713][INFO][o.e.c.s.ClusterApplierService] [h1-es03] added {{h1-es02}{qgmMV2UbT-ScN9uRr6YM8g}{klj1K8UMRHGtYpYozIuqsA}{192.168.57.102}{192.168.57.102:9300}{dimr}}, term: 87, version: 236730, reason: Publication{term=8 7, version=236730} ``` 3 nodes in a cluster, each can be a master and is a datanode. Elasticsearch 7.9.1 build opendistro from amazon. In addition, we write to Elasticsearch using Apache Metron and when we try to write sometimes, we get errors: ```auto java.io.IOException: listener timeout after waiting for [30000] ms ``` ---
**Author:** ![tony](https://avatars.discourse-cdn.com/v4/letter/t/ee7513/32.png) [@tony](https://forum.opensearch.org/u/tony)\ **Post date:** [November 25, 2020, 8:45am UTC](https://forum.opensearch.org/t/nodes-fall-out-of-the-cluster-es-7-9-1/4259/2 "2020-11-25T08:45:12Z")
Do you have the logs for h1-es02 at that time to see what happened with the node? ---
**Author:** ![ThreatInter](https://avatars.discourse-cdn.com/v4/letter/t/49beb7/32.png) [@ThreatInter](https://forum.opensearch.org/u/ThreatInter)\ **Post date:** [December 14, 2020, 6:07am UTC](https://forum.opensearch.org/t/nodes-fall-out-of-the-cluster-es-7-9-1/4259/3 "2020-12-14T06:07:25Z")
We faced with this problem again. Now I have log, but can’t understang how to load it to this forum. Seems that I can’t change my initial message, and can’t attach to new message any file that not a picture. ---
**Author:** ![tmchang](https://avatars.discourse-cdn.com/v4/letter/t/f17d59/32.png) [@tmchang](https://forum.opensearch.org/u/tmchang)\ **Post date:** [March 25, 2021, 3:46pm UTC](https://forum.opensearch.org/t/nodes-fall-out-of-the-cluster-es-7-9-1/4259/4 "2021-03-25T15:46:40Z")
We had the same situation. The ES version is 7.10.2. The master node show “data node node-left”, but the data node show “master not discovered yet” Server layout 172.16.22.153 esnode1 172.16.22.154 esnode2 172.16.22.155 esnode3 172.16.22.190 esnode4 172.16.22.191 esnode5 172.16.22.192 esnode6 172.16.22.193 esnode7 172.16.22.194 esnode8 172.16.22.195 esnode9 Related logs: Master node(esnode1) [2021-03-25T13:58:31,547][INFO][o.e.c.c.C.CoordinatorPublication] [esnode1] after [10s] publication of cluster state version [4502] is still waiting for {esnode5}{RPzC\_iENSOiSynpEvT0zag}{T4D5QV6jRvu43I\_puZ2iXA}{.191}{172.16.22.191:9300}{dir} [SENT\_PUBLISH\_REQUEST], {esnode6}{MsHrFuhtR2yp0JGSRsqS5w}{lA8\_OIWuQm2gZxdXsEDEdA}{172.16.22.192}{172.16.22.192:9300}{dir} [SENT\_PUBLISH\_REQUEST], {esnode7}{Yj-61cgOQ–50f8cila67Q}{BSFsIQiSSKSPFUCQ5g65dQ}{172.16.22.193}{172.16.22.193:9300}{dir} [SENT\_PUBLISH\_REQUEST], {esnode4}{1jH5VI7PQbuLNVWtxhvw8Q}{RpBC6FawS82nBTCjvkh9LQ}{172.16.22.190}{172.16.22.190:9300}{dir} [SENT\_PUBLISH\_REQUEST] [2021-03-25T13:58:51,549][WARN][o.e.c.c.C.CoordinatorPublication] [esnode1] after [30s] publication of cluster state version [4502] is still waiting for {esnode5}{RPzC\_iENSOiSynpEvT0zag}{T4D5QV6jRvu43I\_puZ2iXA}{172.16.22.191}{172.16.22.191:9300}{dir} [SENT\_PUBLISH\_REQUEST], {esnode6}{MsHrFuhtR2yp0JGSRsqS5w}{lA8\_OIWuQm2gZxdXsEDEdA}{172.16.22.192}{172.16.22.192:9300}{dir} [SENT\_PUBLISH\_REQUEST], {esnode7}{Yj-61cgOQ–50f8cila67Q}{BSFsIQiSSKSPFUCQ5g65dQ}{172.16.22.193}{172.16.22.193:9300}{dir} [SENT\_PUBLISH\_REQUEST], {esnode4}{1jH5VI7PQbuLNVWtxhvw8Q}{RpBC6FawS82nBTCjvkh9LQ}{172.16.22.190}{172.16.22.190:9300}{dir} [SENT\_PUBLISH\_REQUEST] [2021-03-25T13:58:51,552][INFO][o.e.c.r.a.AllocationService] [esnode1] updating number\_of\_replicas to [4] for indices [.opendistro\_security] [2021-03-25T13:58:51,556][INFO][o.e.c.s.MasterService] [esnode1] node-left[{esnode8}{HeEjBS5JSCSYeP2zr2MPWA}{nHAb6LFYT6-YjtXO72CO2g}{172.16.22.194}{172.16.22.194:9300}{dir} reason: followers check retry count exceeded], term: 316, version: 4503, delta: removed {{esnode8}{HeEjBS5JSCSYeP2zr2MPWA}{nHAb6LFYT6-YjtXO72CO2g}{172.16.22.194}{172.16.22.194:9300}{dir}} [2021-03-25T13:59:01,558][INFO][o.e.c.c.C.CoordinatorPublication] [esnode1] after [10s] publication of cluster state version [4503] is still waiting for {esnode5}{RPzC\_iENSOiSynpEvT0zag}{T4D5QV6jRvu43I\_puZ2iXA}{172.16.22.191}{172.16.22.191:9300}{dir} [SENT\_PUBLISH\_REQUEST], {esnode6}{MsHrFuhtR2yp0JGSRsqS5w}{lA8\_OIWuQm2gZxdXsEDEdA}{172.16.22.192}{172.16.22.192:9300}{dir} [SENT\_PUBLISH\_REQUEST], {esnode7}{Yj-61cgOQ–50f8cila67Q}{BSFsIQiSSKSPFUCQ5g65dQ}{172.16.22.193}{172.16.22.193:9300}{dir} [SENT\_PUBLISH\_REQUEST], {esnode4}{1jH5VI7PQbuLNVWtxhvw8Q}{RpBC6FawS82nBTCjvkh9LQ}{172.16.22.190}{172.16.22.190:9300}{dir} [SENT\_PUBLISH\_REQUEST] [2021-03-25T13:59:21,558][INFO][o.e.c.s.ClusterApplierService] [esnode1] removed {{esnode8}{HeEjBS5JSCSYeP2zr2MPWA}{nHAb6LFYT6-YjtXO72CO2g}{172.16.22.194}{172.16.22.194:9300}{dir}}, term: 316, version: 4503, reason: Publication{term=316, version=4503} Data node(esnode5) [2021-03-25T14:00:24,436][INFO][o.e.c.c.Coordinator] [esnode5] master node [{esnode1}{raDLHjOiTYaY\_5ckIjnLVA}{VlAg-gG5Q72y0KTORWm-uQ}{172.16.22.153}{172.16.22.153:9300}{imr}] failed, restarting discovery org.elasticsearch.ElasticsearchException: node [{esnode1}{raDLHjOiTYaY\_5ckIjnLVA}{VlAg-gG5Q72y0KTORWm-uQ}{172.16.22.153}{172.16.22.153:9300}{imr}] failed [3] consecutive checks at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:293) ~[elasticsearch-7.10.2.jar:7.10.2] Caused by: org.elasticsearch.transport.RemoteTransportException: [esnode1][172.16.22.153:9300][internal:coordination/fault\_detection/leader\_check] Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: rejecting leader check since [{esnode5}{RPzC\_iENSOiSynpEvT0zag}{T4D5QV6jRvu43I\_puZ2iXA}{172.16.22.191}{172.16.22.191:9300}{dir}] has been removed from the cluster Pls help identify root cause. Thanks, TM ---
**Author:** ![tmchang](https://avatars.discourse-cdn.com/v4/letter/t/f17d59/32.png) [@tmchang](https://forum.opensearch.org/u/tmchang)\ **Post date:** [March 29, 2021, 2:40pm UTC](https://forum.opensearch.org/t/nodes-fall-out-of-the-cluster-es-7-9-1/4259/5 "2021-03-29T14:40:10Z")
Update this issue. We found another post from github opendistro security issue #378. The same situation, user login using AD, then node crash. Reference below link. > > > Learned about the undocumented parameter \`max\_nested\_depth\` for LDAP authorizati…on configuration from opendistro-for-elasticsearch/security#887 > > This PR documents this parameter. > > By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. FYR. TM