Log are hard to understand ( for master node changed )

General Feedback
1

Currently I have this log

[2021-06-10T10:10:07,074][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [adt-sys-es-platform-master-49-177] Exception during establishing a SSL connection: java.io.IOException: Connection timed out
java.io.IOException: Connection timed out
	at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:?]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:47) ~[?:?]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:233) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:389) ~[?:?]
	at org.elasticsearch.transport.CopyBytesSocketChannel.readFromSocketChannel(CopyBytesSocketChannel.java:141) ~[transport-netty4-client-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.CopyBytesSocketChannel.doReadBytes(CopyBytesSocketChannel.java:126) ~[transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-06-10T10:10:07,076][INFO ][o.e.c.c.Coordinator      ] [adt-sys-es-platform-master-49-177] master node [{adt-sys-es-platform-master-51-132}{Za5w1TgdQvex5q1Wh9DcXQ}{s3ImWHgETkO3qj3we3zkVw}{10.3.51.132}{10.3.51.132:9300}{mr}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [adt-sys-es-platform-master-51-132][10.3.51.132:9300][disconnected] disconnected
[2021-06-10T10:10:07,076][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [adt-sys-es-platform-master-49-177] Exception during establishing a SSL connection: java.io.IOException: Connection timed out
java.io.IOException: Connection timed out
	at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[?:?]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:47) ~[?:?]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:233) ~[?:?]
	at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:389) ~[?:?]
	at org.elasticsearch.transport.CopyBytesSocketChannel.readFromSocketChannel(CopyBytesSocketChannel.java:141) ~[transport-netty4-client-7.10.2.jar:7.10.2]
	at org.elasticsearch.transport.CopyBytesSocketChannel.doReadBytes(CopyBytesSocketChannel.java:126) ~[transport-netty4-client-7.10.2.jar:7.10.2]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-06-10T10:10:07,079][INFO ][o.e.c.s.ClusterApplierService] [adt-sys-es-platform-master-49-177] master node changed {previous [{adt-sys-es-platform-master-51-132}{Za5w1TgdQvex5q1Wh9DcXQ}{s3ImWHgETkO3qj3we3zkVw}{10.3.51.132}{10.3.51.132:9300}{mr}], current []}, term: 10, version: 11326, reason: becoming candidate: onLeaderFailure
[2021-06-10T10:10:07,326][INFO ][o.e.c.s.ClusterApplierService] [adt-sys-es-platform-master-49-177] master node changed {previous [], current [{adt-sys-es-platform-master-51-132}{Za5w1TgdQvex5q1Wh9DcXQ}{s3ImWHgETkO3qj3we3zkVw}{10.3.51.132}{10.3.51.132:9300}{mr}]}, term: 10, version: 11328, reason: ApplyCommitRequest{term=10, version=11328, sourceNode={adt-sys-es-platform-master-51-132}{Za5w1TgdQvex5q1Wh9DcXQ}{s3ImWHgETkO3qj3we3zkVw}{10.3.51.132}{10.3.51.132:9300}{mr}}

I have 3 master nodes and 5 data nodes
As far as I understand. Somehow 2 other master nodes can not connect to port 9300 of the current master. Then they decided to change master to another node.
But what I’m missing since I did telnet fine to port 9300 from 2 other master role :sweat: