Un-Assigned Shards after the restart of the ES cluster on Kubernetes

I am using Elasticsearch 7.10.2 deployed on k8s cluster. The master and data nodes are backed by PVs for persisting the data. I have created a Cluster of 3 master and 3 data nodes, with 2 client nodes dedicated. My problem is , when I ingest some data into the data base and re-create the Elasticsearch Cluster. [ the Data , Master nodes are deleted and recreated [ pods ] , then the master and data nodes latches to the same PVs with which it has associated last time ]. After re-creating the cluster the SHARD are shown as UN-ASSIGNED, with the reason ‘Node Left’.

So, can you clarify me , when we are preserving the data on the PVs. Why it is showing ‘UN-ASSIGNED’ even the data/master nodes latching to the same PVs. My worry is in k8s environment then how can we make the data stateful.

logs of the master nodes as follows:

master-0

Starting Elasticsearch:
Java version:
openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment Zulu11.45+52-SA (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.45+52-SA (build 11.0.10+9-LTS, mixed mode)
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-02-07T11:56:08,850][INFO ][o.e.n.Node               ] [elasticsearch-master-0] version[7.10.2], pid[1], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.18.0-348.2.1.el8_5.x86_64/amd64], JVM[Azul Systems, Inc./OpenJDK 64-Bit Server VM/11.0.10/11.0.10+9-LTS]
[2022-02-07T11:56:08,854][INFO ][o.e.n.Node               ] [elasticsearch-master-0] JVM home [/usr/lib/jvm/zre-11], using bundled JDK [false]
[2022-02-07T11:56:08,854][INFO ][o.e.n.Node               ] [elasticsearch-master-0] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-9472843382414815717, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3g, -Xmx3g, -XX:MaxDirectMemorySize=1610612736, -Des.path.home=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2, -Des.path.conf=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-02-07T11:56:10,345][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-0] Loaded scheduler extension: Alias/Rollup, index: .vsa-metricservice-jobs
[2022-02-07T11:56:10,346][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-02-07T11:56:10,351][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [aggs-matrix-stats]
[2022-02-07T11:56:10,351][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [analysis-common]
[2022-02-07T11:56:10,352][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [geo]
[2022-02-07T11:56:10,352][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [ingest-common]
[2022-02-07T11:56:10,353][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [ingest-geoip]
[2022-02-07T11:56:10,353][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [ingest-user-agent]
[2022-02-07T11:56:10,353][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [kibana]
[2022-02-07T11:56:10,354][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [lang-expression]
[2022-02-07T11:56:10,354][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [lang-mustache]
[2022-02-07T11:56:10,354][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [lang-painless]
[2022-02-07T11:56:10,355][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [mapper-extras]
[2022-02-07T11:56:10,355][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [parent-join]
[2022-02-07T11:56:10,355][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [percolator]
[2022-02-07T11:56:10,356][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [rank-eval]
[2022-02-07T11:56:10,356][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [reindex]
[2022-02-07T11:56:10,357][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [repository-url]
[2022-02-07T11:56:10,357][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded module [transport-netty4]
[2022-02-07T11:56:10,358][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded plugin [opendistro-index-management]
[2022-02-07T11:56:10,358][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded plugin [opendistro-job-scheduler]
[2022-02-07T11:56:10,359][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded plugin [repository-s3]
[2022-02-07T11:56:10,359][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-0] loaded plugin [vmware-es-utils-extension]
[2022-02-07T11:56:10,399][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-0] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/rhel-root)]], net usable_space [28.6gb], net total_space [48.6gb], types [xfs]
[2022-02-07T11:56:10,399][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-0] heap size [2.9gb], compressed ordinary object pointers [true]
[2022-02-07T11:56:10,507][INFO ][o.e.n.Node               ] [elasticsearch-master-0] node name [elasticsearch-master-0], node ID [0NGeq5GrRD6gG6BdgKQH7A], cluster name [smartscluster], roles [master, remote_cluster_client]
[2022-02-07T11:56:14,365][INFO ][o.e.t.NettyAllocator     ] [elasticsearch-master-0] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=false, g1gc_region_size=0b}]
[2022-02-07T11:56:14,440][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-master-0] using discovery type [zen] and seed hosts providers [settings]
[2022-02-07T11:56:14,826][WARN ][o.e.g.DanglingIndicesState] [elasticsearch-master-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-02-07T11:56:15,061][INFO ][o.e.n.Node               ] [elasticsearch-master-0] initialized
[2022-02-07T11:56:15,062][INFO ][o.e.n.Node               ] [elasticsearch-master-0] starting ...
[2022-02-07T11:56:15,186][INFO ][o.e.t.TransportService   ] [elasticsearch-master-0] publish_address {10.233.90.62:9300}, bound_addresses {0.0.0.0:9300}
[2022-02-07T11:56:15,266][WARN ][o.e.t.TcpTransport       ] [elasticsearch-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.90.62:9300, remoteAddress=/10.233.96.102:52548}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:952) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:164) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Unknown Source) [?:?]
[2022-02-07T11:56:15,481][WARN ][o.e.t.TcpTransport       ] [elasticsearch-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.90.62:9300, remoteAddress=/10.233.92.86:42818}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:952) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:164) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Unknown Source) [?:?]
[2022-02-07T11:56:15,497][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-02-07T11:56:15,501][INFO ][o.e.c.c.Coordinator      ] [elasticsearch-master-0] cluster UUID [EyjIAk_aTHmj8K7a22E9nw]
[2022-02-07T11:56:19,124][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] master node changed {previous [], current [{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}]}, added {{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}, term: 5, version: 2463, reason: ApplyCommitRequest{term=5, version=2463, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:19,142][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-master-0] publish_address {10.233.90.62:9200}, bound_addresses {0.0.0.0:9200}
[2022-02-07T11:56:19,143][INFO ][o.e.n.Node               ] [elasticsearch-master-0] started
[2022-02-07T11:56:19,344][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir}}, term: 5, version: 2465, reason: ApplyCommitRequest{term=5, version=2465, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:19,557][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir}}, term: 5, version: 2466, reason: ApplyCommitRequest{term=5, version=2466, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:23,358][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-master-2}{GaxnS852Q2yH78QyfsscBA}{xFUOLvxPQyqjEfmHhEZ9TA}{10.233.92.87}{10.233.92.87:9300}{mr}}, term: 5, version: 2467, reason: ApplyCommitRequest{term=5, version=2467, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:59,771][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple}}, term: 5, version: 2468, reason: ApplyCommitRequest{term=5, version=2468, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:59,885][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple}}, term: 5, version: 2469, reason: ApplyCommitRequest{term=5, version=2469, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:57:00,177][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple}}, term: 5, version: 2470, reason: ApplyCommitRequest{term=5, version=2470, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}

logs for second master node. ‘master-1’

Starting Elasticsearch:
Java version:
openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment Zulu11.45+52-SA (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.45+52-SA (build 11.0.10+9-LTS, mixed mode)
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-02-07T11:56:13,519][INFO ][o.e.n.Node               ] [elasticsearch-master-1] version[7.10.2], pid[1], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.18.0-348.2.1.el8_5.x86_64/amd64], JVM[Azul Systems, Inc./OpenJDK 64-Bit Server VM/11.0.10/11.0.10+9-LTS]
[2022-02-07T11:56:13,523][INFO ][o.e.n.Node               ] [elasticsearch-master-1] JVM home [/usr/lib/jvm/zre-11], using bundled JDK [false]
[2022-02-07T11:56:13,524][INFO ][o.e.n.Node               ] [elasticsearch-master-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-9633479102114016921, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3g, -Xmx3g, -XX:MaxDirectMemorySize=1610612736, -Des.path.home=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2, -Des.path.conf=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-02-07T11:56:14,968][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-1] Loaded scheduler extension: Alias/Rollup, index: .vsa-metricservice-jobs
[2022-02-07T11:56:14,969][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-02-07T11:56:14,973][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [aggs-matrix-stats]
[2022-02-07T11:56:14,973][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [analysis-common]
[2022-02-07T11:56:14,973][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [geo]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [ingest-common]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [ingest-geoip]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [ingest-user-agent]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [kibana]
[2022-02-07T11:56:14,974][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-expression]
[2022-02-07T11:56:14,975][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-mustache]
[2022-02-07T11:56:14,975][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [lang-painless]
[2022-02-07T11:56:14,975][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [mapper-extras]
[2022-02-07T11:56:14,975][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [parent-join]
[2022-02-07T11:56:14,976][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [percolator]
[2022-02-07T11:56:14,976][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [rank-eval]
[2022-02-07T11:56:14,976][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [reindex]
[2022-02-07T11:56:14,976][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [repository-url]
[2022-02-07T11:56:14,977][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded module [transport-netty4]
[2022-02-07T11:56:14,977][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded plugin [opendistro-index-management]
[2022-02-07T11:56:14,978][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded plugin [opendistro-job-scheduler]
[2022-02-07T11:56:14,978][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded plugin [repository-s3]
[2022-02-07T11:56:14,978][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-1] loaded plugin [vmware-es-utils-extension]
[2022-02-07T11:56:15,014][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/rhel-root)]], net usable_space [26.5gb], net total_space [48.6gb], types [xfs]
[2022-02-07T11:56:15,015][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-1] heap size [2.9gb], compressed ordinary object pointers [true]
[2022-02-07T11:56:15,112][INFO ][o.e.n.Node               ] [elasticsearch-master-1] node name [elasticsearch-master-1], node ID [auRjv5lkT3iDaCtvM7_txA], cluster name [smartscluster], roles [master, remote_cluster_client]
[2022-02-07T11:56:18,670][INFO ][o.e.t.NettyAllocator     ] [elasticsearch-master-1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=false, g1gc_region_size=0b}]
[2022-02-07T11:56:18,744][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-master-1] using discovery type [zen] and seed hosts providers [settings]
[2022-02-07T11:56:19,084][WARN ][o.e.g.DanglingIndicesState] [elasticsearch-master-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-02-07T11:56:19,280][INFO ][o.e.n.Node               ] [elasticsearch-master-1] initialized
[2022-02-07T11:56:19,281][INFO ][o.e.n.Node               ] [elasticsearch-master-1] starting ...
[2022-02-07T11:56:19,415][INFO ][o.e.t.TransportService   ] [elasticsearch-master-1] publish_address {10.233.105.88:9300}, bound_addresses {0.0.0.0:9300}
[2022-02-07T11:56:19,521][WARN ][o.e.t.TcpTransport       ] [elasticsearch-master-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.105.88:9300, remoteAddress=/10.233.90.62:56514}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
	at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:952) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:164) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Unknown Source) [?:?]
[2022-02-07T11:56:19,685][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-master-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-02-07T11:56:19,688][INFO ][o.e.c.c.Coordinator      ] [elasticsearch-master-1] cluster UUID [EyjIAk_aTHmj8K7a22E9nw]
[2022-02-07T11:56:19,981][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] elected-as-master ([2] nodes joined)[{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr} elect leader, {elasticsearch-master-0}{0NGeq5GrRD6gG6BdgKQH7A}{_arbQfaIR-eZlQAoNyNmGw}{10.233.90.62}{10.233.90.62:9300}{mr} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 5, version: 2463, delta: master node changed {previous [], current [{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}]}, added {{elasticsearch-master-0}{0NGeq5GrRD6gG6BdgKQH7A}{_arbQfaIR-eZlQAoNyNmGw}{10.233.90.62}{10.233.90.62:9300}{mr}}
[2022-02-07T11:56:20,084][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] master node changed {previous [], current [{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}]}, added {{elasticsearch-master-0}{0NGeq5GrRD6gG6BdgKQH7A}{_arbQfaIR-eZlQAoNyNmGw}{10.233.90.62}{10.233.90.62:9300}{mr}}, term: 5, version: 2463, reason: Publication{term=5, version=2463}
[2022-02-07T11:56:20,141][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-master-1] publish_address {10.233.105.88:9200}, bound_addresses {0.0.0.0:9200}
[2022-02-07T11:56:20,141][INFO ][o.e.n.Node               ] [elasticsearch-master-1] started
[2022-02-07T11:56:20,210][INFO ][o.e.g.GatewayService     ] [elasticsearch-master-1] recovered [6] indices into cluster_state
[2022-02-07T11:56:20,230][ERROR][c.a.o.i.i.ManagedIndexCoordinator] [elasticsearch-master-1] get managed-index failed: NoShardAvailableActionException[No shard available for [org.elasticsearch.action.get.MultiGetShardRequest@e10609a]]
[2022-02-07T11:56:20,250][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir} join existing leader], term: 5, version: 2465, delta: added {{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir}}
[2022-02-07T11:56:20,271][ERROR][c.a.o.i.i.ManagedIndexCoordinator] [elasticsearch-master-1] Failed to get ISM templates: Failed to execute phase [query], all shards failed
[2022-02-07T11:56:20,313][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir}}, term: 5, version: 2465, reason: Publication{term=5, version=2465}
[2022-02-07T11:56:20,481][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir} join existing leader], term: 5, version: 2466, delta: added {{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir}}
[2022-02-07T11:56:20,538][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir}}, term: 5, version: 2466, reason: Publication{term=5, version=2466}
[2022-02-07T11:56:24,288][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-master-2}{GaxnS852Q2yH78QyfsscBA}{xFUOLvxPQyqjEfmHhEZ9TA}{10.233.92.87}{10.233.92.87:9300}{mr} join existing leader], term: 5, version: 2467, delta: added {{elasticsearch-master-2}{GaxnS852Q2yH78QyfsscBA}{xFUOLvxPQyqjEfmHhEZ9TA}{10.233.92.87}{10.233.92.87:9300}{mr}}
[2022-02-07T11:56:24,401][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-master-2}{GaxnS852Q2yH78QyfsscBA}{xFUOLvxPQyqjEfmHhEZ9TA}{10.233.92.87}{10.233.92.87:9300}{mr}}, term: 5, version: 2467, reason: Publication{term=5, version=2467}
[2022-02-07T11:57:00,696][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple} join existing leader], term: 5, version: 2468, delta: added {{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple}}
[2022-02-07T11:57:00,800][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple}}, term: 5, version: 2468, reason: Publication{term=5, version=2468}
[2022-02-07T11:57:00,803][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple} join existing leader], term: 5, version: 2469, delta: added {{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple}}
[2022-02-07T11:57:01,103][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple}}, term: 5, version: 2469, reason: Publication{term=5, version=2469}
[2022-02-07T11:57:01,106][INFO ][o.e.c.s.MasterService    ] [elasticsearch-master-1] node-join[{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple} join existing leader], term: 5, version: 2470, delta: added {{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple}}
[2022-02-07T11:57:01,402][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-1] added {{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple}}, term: 5, version: 2470, reason: Publication{term=5, version=2470}
[2022-02-07T12:01:19,692][INFO ][c.a.o.j.s.JobSweeper     ] [elasticsearch-master-1] Running full sweep

Logs for the third master node ‘master-2’

Starting Elasticsearch:
Java version:
openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment Zulu11.45+52-SA (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.45+52-SA (build 11.0.10+9-LTS, mixed mode)
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-02-07T11:56:17,803][INFO ][o.e.n.Node               ] [elasticsearch-master-2] version[7.10.2], pid[1], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.18.0-348.2.1.el8_5.x86_64/amd64], JVM[Azul Systems, Inc./OpenJDK 64-Bit Server VM/11.0.10/11.0.10+9-LTS]
[2022-02-07T11:56:17,808][INFO ][o.e.n.Node               ] [elasticsearch-master-2] JVM home [/usr/lib/jvm/zre-11], using bundled JDK [false]
[2022-02-07T11:56:17,808][INFO ][o.e.n.Node               ] [elasticsearch-master-2] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-12968836566142102889, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3g, -Xmx3g, -XX:MaxDirectMemorySize=1610612736, -Des.path.home=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2, -Des.path.conf=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-02-07T11:56:19,165][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-2] Loaded scheduler extension: Alias/Rollup, index: .vsa-metricservice-jobs
[2022-02-07T11:56:19,166][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-2] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-02-07T11:56:19,170][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [aggs-matrix-stats]
[2022-02-07T11:56:19,170][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [analysis-common]
[2022-02-07T11:56:19,171][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [geo]
[2022-02-07T11:56:19,171][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [ingest-common]
[2022-02-07T11:56:19,171][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [ingest-geoip]
[2022-02-07T11:56:19,171][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [ingest-user-agent]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [kibana]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [lang-expression]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [lang-mustache]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [lang-painless]
[2022-02-07T11:56:19,172][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [mapper-extras]
[2022-02-07T11:56:19,173][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [parent-join]
[2022-02-07T11:56:19,173][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [percolator]
[2022-02-07T11:56:19,173][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [rank-eval]
[2022-02-07T11:56:19,173][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [reindex]
[2022-02-07T11:56:19,174][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [repository-url]
[2022-02-07T11:56:19,174][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded module [transport-netty4]
[2022-02-07T11:56:19,174][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded plugin [opendistro-index-management]
[2022-02-07T11:56:19,175][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded plugin [opendistro-job-scheduler]
[2022-02-07T11:56:19,175][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded plugin [repository-s3]
[2022-02-07T11:56:19,175][INFO ][o.e.p.PluginsService     ] [elasticsearch-master-2] loaded plugin [vmware-es-utils-extension]
[2022-02-07T11:56:19,214][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-2] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/rhel-root)]], net usable_space [15.1gb], net total_space [48.6gb], types [xfs]
[2022-02-07T11:56:19,214][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-master-2] heap size [2.9gb], compressed ordinary object pointers [true]
[2022-02-07T11:56:19,313][INFO ][o.e.n.Node               ] [elasticsearch-master-2] node name [elasticsearch-master-2], node ID [GaxnS852Q2yH78QyfsscBA], cluster name [smartscluster], roles [master, remote_cluster_client]
[2022-02-07T11:56:22,951][INFO ][o.e.t.NettyAllocator     ] [elasticsearch-master-2] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=false, g1gc_region_size=0b}]
[2022-02-07T11:56:23,027][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-master-2] using discovery type [zen] and seed hosts providers [settings]
[2022-02-07T11:56:23,364][WARN ][o.e.g.DanglingIndicesState] [elasticsearch-master-2] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-02-07T11:56:23,620][INFO ][o.e.n.Node               ] [elasticsearch-master-2] initialized
[2022-02-07T11:56:23,620][INFO ][o.e.n.Node               ] [elasticsearch-master-2] starting ...
[2022-02-07T11:56:23,747][INFO ][o.e.t.TransportService   ] [elasticsearch-master-2] publish_address {10.233.92.87:9300}, bound_addresses {0.0.0.0:9300}
[2022-02-07T11:56:24,032][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-master-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-02-07T11:56:24,035][INFO ][o.e.c.c.Coordinator      ] [elasticsearch-master-2] cluster UUID [EyjIAk_aTHmj8K7a22E9nw]
[2022-02-07T11:56:24,336][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] master node changed {previous [], current [{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}]}, added {{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir},{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr},{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir},{elasticsearch-master-0}{0NGeq5GrRD6gG6BdgKQH7A}{_arbQfaIR-eZlQAoNyNmGw}{10.233.90.62}{10.233.90.62:9300}{mr}}, term: 5, version: 2467, reason: ApplyCommitRequest{term=5, version=2467, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:24,387][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-master-2] publish_address {10.233.92.87:9200}, bound_addresses {0.0.0.0:9200}
[2022-02-07T11:56:24,387][INFO ][o.e.n.Node               ] [elasticsearch-master-2] started
[2022-02-07T11:57:00,703][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] added {{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple}}, term: 5, version: 2468, reason: ApplyCommitRequest{term=5, version=2468, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:57:00,802][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] added {{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple}}, term: 5, version: 2469, reason: ApplyCommitRequest{term=5, version=2469, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:57:01,107][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-2] added {{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple}}, term: 5, version: 2470, reason: ApplyCommitRequest{term=5, version=2470, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T12:01:24,038][INFO ][c.a.o.j.s.JobSweeper     ] [elasticsearch-master-2] Running full sweep