Trying to create a Opensearch Public Cluster using 3 Public IP Addresses is Failing

Community
,
1

Logs are from 10.10.10.10 server which tries to connect to itself and also another ip - 20.20.20.20 to form the cluster

  1. Caused by: java.io.IOException: Connection refused: 10.10.10.10/10.10.10.10:9300
  2. [2023-09-12T14:23:48,746][INFO ][o.o.c.c.JoinHelper ] [10.10.10.10] failed to join {20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.20.20.20}{20.20.20.20:9300}
  3. Caused by: org.opensearch.transport.ConnectTransportException: [10.10.10.10][10.10.10.10:9300] handshake_timeout[30s]

----------------------Log Exhibit 1--------------

[2023-09-12T14:23:48,689][DEBUG][o.o.c.c.JoinHelper ] [10.10.10.10] attempting to join {20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.21
9.68.198}{20.20.20.20:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w
}{10.10.10.10}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=163, optionalJoin=Optional[Join{term=163, lastAcceptedTerm=0, lastAccep
tedVersion=0, sourceNode={10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w}{10.10.10.10}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true
}, targetNode={20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.20.20.20}{20.20.20.20:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
[2023-09-12T14:23:48,690][DEBUG][o.o.t.TransportService ] [10.10.10.10] Action: internal:cluster/coordination/join
[2023-09-12T14:23:48,739][INFO ][o.o.c.c.JoinHelper ] [10.10.10.10] failed to join {20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.219.68
.198}{20.20.20.20:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w}{14
.98.59.203}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=163, optionalJoin=Optional[Join{term=163, lastAcceptedTerm=0, lastAcceptedV
ersion=0, sourceNode={10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w}{10.10.10.10}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true}, t
argetNode={20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.20.20.20}{20.20.20.20:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [20.20.20.20][172.18.0.6:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [10.10.10.10][10.10.10.10:9300] connect_exception
at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1076) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.action.ActionListener.lambda$toBiConsumer$2(ActionListener.java:215) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.concurrent.CompletableContext.lambda$addListener$0(CompletableContext.java:57) ~[opensearch-common-2.8.0.jar:2.8.0]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
at org.opensearch.common.concurrent.CompletableContext.completeExceptionally(CompletableContext.java:72) ~[opensearch-common-2.8.0.jar:2.8.0]
at org.opensearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:81) ~[transport-netty4-client-2.8.0.jar:2.8.0]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.91.Final.jar:4.1.91.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.91.Final.jar:4.1.91.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.91.Final.jar:4.1.91.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.91.Final.jar:4.1.91.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.91.Final.jar:4.1.91.Final]
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[netty-common-4.1.91.Final.jar:4.1.91.Final]
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[netty-common-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[netty-transport-4.1.91.Final.jar:4.1.91.F
inal]
at at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.io.IOException: Connection refused: 10.10.10.10/10.10.10.10:9300
Caused by: java.io.IOException: Connection refused
at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
at sun.nio.ch.Net.pollConnectNow(Net.java:672) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946) ~[?:?]
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2023-09-12T14:23:48,746][INFO ][o.o.c.c.JoinHelper ] [10.10.10.10] failed to join {20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.20.20.20}{20.20.20.20:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w}{10.10.10.10}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=163, optionalJoin=Optional[Join{term=163, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w}{10.10.10.10}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.20.20.20}{20.20.20.20:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [20.20.20.20][172.18.0.6:9300][internal:cluster/coordination/join]
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) [netty-transport-4.1.91.Final.jar:4.1.91.Final]

---------------Log Exhibit 2------------

[2023-09-12T14:47:58,585][WARN ][o.o.c.c.JoinHelper ] [10.10.10.10] last failed join attempt was 4.4s ago, failed to join {20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.20.20.20}{20.20.20.20:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w}{10.10.10.10}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=164, optionalJoin=Optional[Join{term=164, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w}{10.10.10.10}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.20.20.20}{20.20.20.20:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [20.20.20.20][172.18.0.6:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [10.10.10.10][10.10.10.10:9300] handshake_timeout[30s]
at org.opensearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:106) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-09-12T14:47:58,586][WARN ][o.o.c.c.ClusterFormationFailureHelper] [10.10.10.10] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [20.20.20.20] to bootstrap a cluster: have discovered [{10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w}{10.10.10.10}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true}, {15.15.15.15}{5J1_8tVrTRe-XMSgDYIc4w}{cz0oLWT0RLyNOrrfukb59w}{15.15.15.15}{15.15.15.15:9300}{dimr}{shard_indexing_pressure_enabled=true}, {20.20.20.20}{_qTpr_tLRrmP3DhNxyNEAg}{P8i_Uc3ASiyCNURHFMEeFw}{20.20.20.20}{20.20.20.20:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [20.20.20.20:9300] from hosts providers and [{10.10.10.10}{cmLCARrGT_2JtUBFpaon8A}{7dZtRZdLQ1Kei_cuvMOC1w}{10.10.10.10}{10.10.10.10:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 164, last-accepted version 0 in term 0

-----------------------------Log Exhibit 3------------------------

[2023-09-12T14:47:57,866][DEBUG][r.suppressed ] [10.10.10.10] path: /bad-request, params: {}
java.lang.IllegalArgumentException: text is empty (possibly HTTP/0.9)
at io.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:72) ~[netty-codec-http-4.1.91.Final.jar:4.1.91.Final]
at io.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:153) ~[netty-codec-http-4.1.91.Final.jar:4.1.91.Final]
at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:277) ~[netty-codec-http-4.1.91.Final.jar:4.1.91.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.91.Final.jar:4.1.91.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) [netty-codec-4.1.91.Final.jar:4.1.91.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) [netty-codec-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.91.Final.jar:4.1.91.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.91.Final.jar:4.1.91.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.91.Final.jar:4.1.91.Final]

2

Hey @deepakdubey

Can you show you config file/s? i

3

IP addressing implies a routing issue between the two servers!

10.10.10.10 is not a public address as per RFC 1918. 10.0.0.0/8 is one of the three IP address ranges not publicly routable, which allows us to use Class C, Class B or Class A address ranges behind a firewall.

Connectivity between 10.0.0.0/8 and 20.20.20.20 requires, at the very least, a router in between the servers with address translation for any IP address to route 10.10.10.10 to/from 20.20.20.20

The IP addressing implies these servers are on different networks and have a firewall/router between them. Perhaps you should be looking at the firewall and/or router rules between them.

If these two addresses are on the same network without a router/firewall between them, then that is the problem.

1 Like
4

Sorry, I had to mask the public ip earlier so I gave 10.10.10.10/20.20.20.20.

“14.98.59.206” and “14.98.59.203” are the 2 public ips which I am trying to create opensearch cluster.

There are 2 public ip addresses “14.98.59.206” and “14.98.59.203” forming part of Cluster.
These nodes are currently behind a NAT Public IP 106.51.82.216
I started “14.98.59.206” first which starts up fine with a single node cluster that can be seen via
curl http://14.98.59.206:9200
curl http://14.98.59.206:9200/_cat/nodes

Both nodes have identical settings except node.name, network.host and network.publish_host
Bidirectional ping, telnet, nc on port 9200,9300 is working fine.

Now when I start node “14.98.59.203” I see errors as can be seen in logs section below:-

— opensearch.yml (from “14.98.59.203”) —
cluster.name: my-application
plugins.security.disabled: true
discovery.probe.connect_timeout: 300s
node.name: 14.98.59.203
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
logger.level: debug
network.host: 14.98.59.203
network.bind_host: 0.0.0.0
network.publish_host: 14.98.59.203
http.port: 9200
discovery.seed_hosts: [“14.98.59.203”,“14.98.59.206”]
cluster.initial_cluster_manager_nodes: [“14.98.59.206”]
transport.tcp.connect_timeout: 54s
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de
    plugins.security.audit.type: internal_opensearch
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
    plugins.security.system_indices.enabled: true
    plugins.security.system_indices.indices: [“.plugins-ml-model-group”, “.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.ql-datasources”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”, “.opensearch-knn-models”]
    node.max_local_storage_nodes: 3

— Logs from “14.98.59.203” —

[2023-09-20T22:51:19,841][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:19,850][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:20,842][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:20,847][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:21,843][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:21,848][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:22,844][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:22,848][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:23,845][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:23,850][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:24,633][WARN ][o.o.c.c.ClusterFormationFailureHelper] [14.98.59.203] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [14.98.59.206] to bootstrap a cluster: have discovered [{14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [14.98.59.206:9300] from hosts providers and [{14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 1, last-accepted version 0 in term 0
[2023-09-20T22:51:24,847][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:24,851][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:25,848][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:25,857][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:26,820][INFO ][o.o.c.c.JoinHelper ] [14.98.59.203] failed to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [14.98.59.206][192.168.1.25:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [14.98.59.203][14.98.59.203:9300] handshake_timeout[54s]
at org.opensearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:106) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-09-20T22:51:26,821][INFO ][o.o.c.c.JoinHelper ] [14.98.59.203] failed to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [14.98.59.206][192.168.1.25:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [14.98.59.203][14.98.59.203:9300] handshake_timeout[54s]
at org.opensearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:106) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-09-20T22:51:34,634][WARN ][o.o.c.c.JoinHelper ] [14.98.59.203] last failed join attempt was 7.8s ago, failed to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [14.98.59.206][192.168.1.25:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [14.98.59.203][14.98.59.203:9300] handshake_timeout[54s]
at org.opensearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:106) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-09-20T22:51:34,635][WARN ][o.o.c.c.ClusterFormationFailureHelper] [14.98.59.203] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [14.98.59.206] to bootstrap a cluster: have discovered [{14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [14.98.59.206:9300] from hosts providers and [{14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 1, last-accepted version 0 in term 0
[2023-09-20T22:51:34,858][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:34,868][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:35,860][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:35,865][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:36,861][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:36,867][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:37,862][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:37,869][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:38,864][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:38,868][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:51:39,865][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:51:39,871][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request

[2023-09-20T22:51:54,638][WARN ][o.o.c.c.ClusterFormationFailureHelper] [14.98.59.203] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [14.98.59.206] to bootstrap a cluster: have discovered [{14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [14.98.59.206:9300] from hosts providers and [{14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 1, last-accepted version 0 in term 0

[2023-09-20T22:52:14,641][WARN ][o.o.c.c.ClusterFormationFailureHelper] [14.98.59.203] cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [14.98.59.206] to bootstrap a cluster: have discovered [{14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}]; discovery will continue using [14.98.59.206:9300] from hosts providers and [{14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 1, last-accepted version 0 in term 0
[2023-09-20T22:52:14,914][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:52:14,922][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:52:15,915][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:52:15,923][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:52:16,916][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:52:16,920][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:52:17,917][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:52:17,922][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:52:18,918][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:52:18,925][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:52:19,920][DEBUG][o.o.t.TransportService ] [14.98.59.203] Action: internal:discovery/request_peers
[2023-09-20T22:52:19,928][DEBUG][o.o.c.c.JoinHelper ] [14.98.59.203] already attempting to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with request JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}, not sending request
[2023-09-20T22:52:20,873][INFO ][o.o.c.c.JoinHelper ] [14.98.59.203] failed to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [14.98.59.206][192.168.1.25:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [14.98.59.203][14.98.59.203:9300] handshake_timeout[54s]
at org.opensearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:106) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2023-09-20T22:52:20,874][INFO ][o.o.c.c.JoinHelper ] [14.98.59.203] failed to join {14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={14.98.59.203}{woCquU14Qh2Fqc5LIEB29w}{m8iwaHfzTT6eVDCOKzY9CA}{14.98.59.203}{14.98.59.203:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={14.98.59.206}{RTqvKUgUTcakz1GM3de9pA}{FOYyiTTTSdWOe-5ZJWcZWw}{14.98.59.206}{14.98.59.206:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
org.opensearch.transport.RemoteTransportException: [14.98.59.206][192.168.1.25:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [14.98.59.203][14.98.59.203:9300] handshake_timeout[54s]
at org.opensearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:106) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]

org.opensearch.transport.RemoteTransportException: [14.98.59.206][192.168.1.25:9300][internal:cluster/coordination/join]
Caused by: org.opensearch.transport.ConnectTransportException: [14.98.59.203][14.98.59.203:9300] handshake_timeout[54s]
at org.opensearch.transport.TransportHandshaker.lambda$sendHandshake$1(TransportHandshaker.java:106) ~[opensearch-2.8.0.jar:2.8.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) ~[opensearch-2.8.0.jar:2.8.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]

5

Hey @deepakdubey

if those configurations are for

what does the configuration for your manager node\ look like.

What Im see in the logs is

cluster-manager not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover cluster-manager-eligible nodes [14.98.59.206] to bootstrap a cluster: have discovered [{14.98.59.203}

EDIT: I just seen this in your logs also

You have a node with address 192.168.1.25? This doesnt match the network of the other nodes.

6

— opensearch.yml (“14.98.59.206”) —

cluster.name: my-application
plugins.security.disabled: true
discovery.probe.connect_timeout: 300s
node.name: 14.98.59.206
path.data: /home/user-hdd/opensearch
path.logs: /var/log/opensearch
logger.level: debug
network.host: 14.98.59.206
network.bind_host: 0.0.0.0
network.publish_host: 14.98.59.206
http.port: 9200
discovery.seed_hosts: [“14.98.59.203”,“14.98.59.206”]
cluster.initial_cluster_manager_nodes: [“14.98.59.206”]
transport.tcp.connect_timeout: 54s
plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • CN=kirk,OU=client,O=client,L=test, C=de
    plugins.security.audit.type: internal_opensearch
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
    plugins.security.system_indices.enabled: true
    plugins.security.system_indices.indices: [“.plugins-ml-model-group”, “.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.ql-datasources”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”, “.opensearch-knn-models”]
    node.max_local_storage_nodes: 3

“192.168.1.25” is the private ip of “14.98.59.206”. I have not specified anywhere the private ips but opensearch has on its own discovered/figured out the private ip.

7

That might be the problem (“192.168.1.25” is the private IP of “14.98.59.206”). Your server has multiple IP addresses, yet the configuration has (“network.bind_host: 0.0.0.0”). You should change the IP address of the network.bind_host with the specific address for your cluster communication. 0.0.0.0 for the bind address translates to the server using all IP addresses to send/receive, the cluster communication might be trying to use any address, public or private.

I’m unsure why you use public addresses instead of private ones for cluster communication. I would recommend private addresses for the cluster communication and a reverse proxy for anything facing the public.

For now, perhaps you can change the network.bind_host address to the specific public address and see if that works :slight_smile:

8

Hey @deepakdubey

I concur with @surfingjoe

Edit: for troubleshooting you could check both nodes with…

telnet {host} {port}

9

telnet {host} {port} works on 9200,9300 from both public ip addresses

10

If
network.bind_host: PUBLIC-IP-ADDRESS
then
Server will not even startup with logs saying unable to bind to PUBLIC-IP-ADDRESS

11

I can confirm 100%

If
network.bind_host: PUBLIC-IP-ADDRESS
then
Even a Single Node Server will not startup (1 Node Cluster)

as it will unable to bind with the PUBLIC IP ADDRESS (which comes from network interface)

I have tested this on 2 different on-premises server and 1 VM on Azure Cloud.

Same Result.

Problem is it cannot bind the public ip address to network interface because usually that comes either from our isolated firewall software or in Azure cloud gets assigned separately.