I am using Elasticsearch 7.10.2 deployed on k8s cluster. The master and data nodes are backed by PVs for persisting the data. I have created a Cluster of 3 master and 3 data nodes, with 2 client nodes dedicated. My problem is , when I ingest some data into the data base and re-create the Elasticsearch Cluster. [ the Data , Master nodes are deleted and recreated [ pods ] , then the master and data nodes latches to the same PVs with which it has associated last time ]. After re-creating the cluster the SHARD are shown as UN-ASSIGNED, with the reason ‘Node Left’.
So, can you clarify me , when we are preserving the data on the PVs. Why it is showing ‘UN-ASSIGNED’ even the data/master nodes latching to the same PVs. My worry is in k8s environment then how can we make the data stateful.
logs of the master nodes as follows:
master-0
Starting Elasticsearch:
Java version:
openjdk version "11.0.10" 2021-01-19 LTS
OpenJDK Runtime Environment Zulu11.45+52-SA (build 11.0.10+9-LTS)
OpenJDK 64-Bit Server VM Zulu11.45+52-SA (build 11.0.10+9-LTS, mixed mode)
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2022-02-07T11:56:08,850][INFO ][o.e.n.Node ] [elasticsearch-master-0] version[7.10.2], pid[1], build[oss/tar/747e1cc71def077253878a59143c1f785afa92b9/2021-01-13T00:42:12.435326Z], OS[Linux/4.18.0-348.2.1.el8_5.x86_64/amd64], JVM[Azul Systems, Inc./OpenJDK 64-Bit Server VM/11.0.10/11.0.10+9-LTS]
[2022-02-07T11:56:08,854][INFO ][o.e.n.Node ] [elasticsearch-master-0] JVM home [/usr/lib/jvm/zre-11], using bundled JDK [false]
[2022-02-07T11:56:08,854][INFO ][o.e.n.Node ] [elasticsearch-master-0] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-9472843382414815717, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms3g, -Xmx3g, -XX:MaxDirectMemorySize=1610612736, -Des.path.home=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2, -Des.path.conf=/usr/share/elasticsearch/opendistroforelasticsearch-1.13.2/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2022-02-07T11:56:10,345][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-0] Loaded scheduler extension: Alias/Rollup, index: .vsa-metricservice-jobs
[2022-02-07T11:56:10,346][INFO ][c.a.o.j.JobSchedulerPlugin] [elasticsearch-master-0] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-02-07T11:56:10,351][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [aggs-matrix-stats]
[2022-02-07T11:56:10,351][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [analysis-common]
[2022-02-07T11:56:10,352][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [geo]
[2022-02-07T11:56:10,352][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [ingest-common]
[2022-02-07T11:56:10,353][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [ingest-geoip]
[2022-02-07T11:56:10,353][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [ingest-user-agent]
[2022-02-07T11:56:10,353][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [kibana]
[2022-02-07T11:56:10,354][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [lang-expression]
[2022-02-07T11:56:10,354][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [lang-mustache]
[2022-02-07T11:56:10,354][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [lang-painless]
[2022-02-07T11:56:10,355][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [mapper-extras]
[2022-02-07T11:56:10,355][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [parent-join]
[2022-02-07T11:56:10,355][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [percolator]
[2022-02-07T11:56:10,356][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [rank-eval]
[2022-02-07T11:56:10,356][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [reindex]
[2022-02-07T11:56:10,357][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [repository-url]
[2022-02-07T11:56:10,357][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded module [transport-netty4]
[2022-02-07T11:56:10,358][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded plugin [opendistro-index-management]
[2022-02-07T11:56:10,358][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded plugin [opendistro-job-scheduler]
[2022-02-07T11:56:10,359][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded plugin [repository-s3]
[2022-02-07T11:56:10,359][INFO ][o.e.p.PluginsService ] [elasticsearch-master-0] loaded plugin [vmware-es-utils-extension]
[2022-02-07T11:56:10,399][INFO ][o.e.e.NodeEnvironment ] [elasticsearch-master-0] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/rhel-root)]], net usable_space [28.6gb], net total_space [48.6gb], types [xfs]
[2022-02-07T11:56:10,399][INFO ][o.e.e.NodeEnvironment ] [elasticsearch-master-0] heap size [2.9gb], compressed ordinary object pointers [true]
[2022-02-07T11:56:10,507][INFO ][o.e.n.Node ] [elasticsearch-master-0] node name [elasticsearch-master-0], node ID [0NGeq5GrRD6gG6BdgKQH7A], cluster name [smartscluster], roles [master, remote_cluster_client]
[2022-02-07T11:56:14,365][INFO ][o.e.t.NettyAllocator ] [elasticsearch-master-0] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=false, g1gc_region_size=0b}]
[2022-02-07T11:56:14,440][INFO ][o.e.d.DiscoveryModule ] [elasticsearch-master-0] using discovery type [zen] and seed hosts providers [settings]
[2022-02-07T11:56:14,826][WARN ][o.e.g.DanglingIndicesState] [elasticsearch-master-0] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-02-07T11:56:15,061][INFO ][o.e.n.Node ] [elasticsearch-master-0] initialized
[2022-02-07T11:56:15,062][INFO ][o.e.n.Node ] [elasticsearch-master-0] starting ...
[2022-02-07T11:56:15,186][INFO ][o.e.t.TransportService ] [elasticsearch-master-0] publish_address {10.233.90.62:9300}, bound_addresses {0.0.0.0:9300}
[2022-02-07T11:56:15,266][WARN ][o.e.t.TcpTransport ] [elasticsearch-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.90.62:9300, remoteAddress=/10.233.96.102:52548}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:952) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:164) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-client-7.10.2.jar:7.10.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Unknown Source) [?:?]
[2022-02-07T11:56:15,481][WARN ][o.e.t.TcpTransport ] [elasticsearch-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.233.90.62:9300, remoteAddress=/10.233.92.86:42818}], closing connection
java.lang.IllegalStateException: transport not ready yet to handle incoming requests
at org.elasticsearch.transport.TransportService.onRequestReceived(TransportService.java:952) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:164) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:89) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:700) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:142) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:117) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:82) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74) [transport-netty4-client-7.10.2.jar:7.10.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Unknown Source) [?:?]
[2022-02-07T11:56:15,497][INFO ][o.e.b.BootstrapChecks ] [elasticsearch-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-02-07T11:56:15,501][INFO ][o.e.c.c.Coordinator ] [elasticsearch-master-0] cluster UUID [EyjIAk_aTHmj8K7a22E9nw]
[2022-02-07T11:56:19,124][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] master node changed {previous [], current [{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}]}, added {{elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}, term: 5, version: 2463, reason: ApplyCommitRequest{term=5, version=2463, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:19,142][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-master-0] publish_address {10.233.90.62:9200}, bound_addresses {0.0.0.0:9200}
[2022-02-07T11:56:19,143][INFO ][o.e.n.Node ] [elasticsearch-master-0] started
[2022-02-07T11:56:19,344][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-client-844bc949d5-q8xfm}{5akk8yAjR0GgJ-kSE79CNQ}{dlaSI7cgTSm5KZbC_FKAoQ}{10.233.96.102}{10.233.96.102:9300}{ir}}, term: 5, version: 2465, reason: ApplyCommitRequest{term=5, version=2465, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:19,557][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-client-844bc949d5-rqzq8}{VuORxXwVRROxFKkMJGsh4A}{IIchPuVfR7Kbltxhl3Ogaw}{10.233.92.86}{10.233.92.86:9300}{ir}}, term: 5, version: 2466, reason: ApplyCommitRequest{term=5, version=2466, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:23,358][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-master-2}{GaxnS852Q2yH78QyfsscBA}{xFUOLvxPQyqjEfmHhEZ9TA}{10.233.92.87}{10.233.92.87:9300}{mr}}, term: 5, version: 2467, reason: ApplyCommitRequest{term=5, version=2467, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:59,771][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-data-2}{tsuG5QNgS12BHVH3GLOspQ}{u-izcjYcTSOqs_iiEp5Hmg}{10.233.92.88}{10.233.92.88:9300}{dr}{group=simple}}, term: 5, version: 2468, reason: ApplyCommitRequest{term=5, version=2468, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:56:59,885][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-data-0}{jqvMza_uR1KlDc79BJSbAw}{lwrRdzT-QxuIYRb0mqImkQ}{10.233.108.88}{10.233.108.88:9300}{dr}{group=simple}}, term: 5, version: 2469, reason: ApplyCommitRequest{term=5, version=2469, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}
[2022-02-07T11:57:00,177][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-master-0] added {{elasticsearch-data-1}{aDxHnQuAT_aenq6wR6HmuQ}{z47s0RpDS16JO1AKd8NJ_w}{10.233.96.105}{10.233.96.105:9300}{dr}{group=simple}}, term: 5, version: 2470, reason: ApplyCommitRequest{term=5, version=2470, sourceNode={elasticsearch-master-1}{auRjv5lkT3iDaCtvM7_txA}{HTeTHejpRpSla14QlHgtuA}{10.233.105.88}{10.233.105.88:9300}{mr}}