org.opensearch.transport.ConnectTransportException: [opensearch-cluster-master-18][10.42.2.20:9300] connect_timeout[30s]

OpenSearch
,
1
  • May I know how opensearch doing peer check? Is it possible to run the same check as NodeConnectionsService to the remote targets(client, data or master) via curl to validate the network or pod condition.
  • Instances are added/removed randomly.
  • Is there known bug in 1.2.4 release for certain symptom? In a small scale setup, not seeing this problem. The problem is observed from a OpenSearch consist of 140 data, 80 masters and 80 clients.

OpenSearch version: 1.2.4

[2022-04-08T08:35:23,752][WARN ][o.o.c.NodeConnectionsService] [opensearch-cluster-master-1] failed to connect to {opensearch-cluster-client-51}{eo7Rd_i8Q-GATVzpQ--zmg}{sJmfxsuGSmOCPwSaTUxD5w}{10.42.4.6}{10.42.4.6:9300}{ir}{shard_indexing_pressure_enabled=true} (tried [1] times)
org.opensearch.transport.ConnectTransportException: [opensearch-cluster-client-51][10.42.4.6:9300] connect_timeout[30s]
at org.opensearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1070) ~[opensearch-1.2.4.jar:1.2.4]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:733) ~[opensearch-1.2.4.jar:1.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]

[2022-04-08T08:05:37,583][INFO ][o.o.c.s.ClusterApplierService] [opensearch-cluster-master-73] added {{opensearch-cluster-client-69}{nigWd0MeSN66jOw3YwPXEA}{qZgQ6wE8TkC8aNxcdtNeMQ}{10.42.4.11}{10.42.4.11:9300}{ir}{shard_indexing_pressure_enabled=true},{opensearch-cluster-client-54}{XucxhUbPRWOtL_OZR0qQ9Q}{YywOjo3dQ5iuDphaPpr8Qg}{10.42.3.206}{10.42.3.206:9300}{ir}{shard_indexing_pressure_enabled=true},{opensearch-cluster-client-70}{9bQn9ukQS5uL6Iv2Udk-ng}{6era5uDjQpO2Xo19mRiBMw}{10.42.6.223}{10.42.6.223:9300}{ir}{shard_indexing_pressure_enabled=true},{opensearch-cluster-client-56}{p4IdoCTZSaqQI5n1NlAOag}{JSLy_WXGTNaF_hlNElYqyQ}{10.42.0.116}{10.42.0.116:9300}{ir}{shard_indexing_pressure_enabled=true}}, term: 304, version: 19993, reason: ApplyCommitRequest{term=304, version=19993, sourceNode={opensearch-cluster-master-61}{fsmqOP7NQ9Ku0Pk3tmRf2g}{fTD_OX6QQTKbNJg7qXOVrw}{10.42.3.187}{10.42.3.187:9300}{m}{shard_indexing_pressure_enabled=true}}


[2022-04-08T08:07:43,336][INFO ][o.o.c.s.ClusterApplierService] [opensearch-cluster-master-73] removed {{opensearch-cluster-client-69}{nigWd0MeSN66jOw3YwPXEA}{qZgQ6wE8TkC8aNxcdtNeMQ}{10.42.4.11}{10.42.4.11:9300}{ir}{shard_indexing_pressure_enabled=true}}, term: 304, version: 19999, reason: ApplyCommitRequest{term=304, version=19999, sourceNode={opensearch-cluster-master-61}{fsmqOP7NQ9Ku0Pk3tmRf2g}{fTD_OX6QQTKbNJg7qXOVrw}{10.42.3.187}{10.42.3.187:9300}{m}{shard_indexing_pressure_enabled=true}}

Screen Shot 2022-04-08 at 5.05.29 PM
Screen Shot 2022-04-08 at 5.05.29 PM1451×381 176 KB

2

I would check your logs for more detail. Your instances might be undersized, and some load event is pushing them over, so they are crashing and restarting, and likewise are leaving and rejoining your cluster. I suspect that is actually what’s happening, here, but can’t say for certain based on the info provided.
Be sure to check your syslog if there isn’t any evidence in opensearch’s log; often there will be some barfage output in your linux distro syslog, and/or systemd’s journal.

Another thought might be that cluster_state is very large, causing timeouts on connection attempt. I saw your host is named master-18… Your cluster doesn’t have 18 masters, does it? The recommended number of masters in a cluster is 3, with a quorum of 2.

Another thought might be that your acting master instance is too loaded, and cannot keep up with heartbeat type things, and nodes are dropping and rejoining for that reason.

Either way, You need to look at more logs. Look for evidence of crashing, or focus on the log from your acting master node.

3

Hi mhoydis,
Thanks for the response. Some more context about this cluster and use case.

  • This cluster is build by helm chart and running in a k8s which consist of 12 powerful physical machines. Each has 12 NVMEs. Each NVME is used as opensearch-data role. So we have around 104 data role nodes ( k8s pods). And there’s 80 opensearch-client pods and 80 openserach-master.
  • 10TB indexed documents per day. That’s around 16 billions documents per day.
green open filebeat-proxy-server-2022.04.07     OK-Y_e0RT-iksJWuj2C6FA 50   1  4,559,306,980  0   5.5tb   2.7tb
green open filebeat-object-server-2022.04.07    uW2SV62rRfqAx14kPrRK_A 50   0 12,049,821,773  0   4.8tb   4.8tb
green open filebeat-object-server-2022.04.08    EIX_9FHqTAqlMrBfJAX1vQ 50   0 12,765,540,385  0   5.1tb   5.1tb

Another thought might be that cluster_state is very large

How to check the cluster_state size?

check your logs for more detail

Unfortunately, these instances are running as k8s pod and there’s no system log. It’s lacking of basic troubleshooting tools inside the image.

Your cluster doesn’t have 18 masters, does it?

well, we had 80 and I lower it to 9 since reading your response.

4

9 is still too many masters. You want exactly 3 masters with a quorum of 2. It could be a reason why your nodes drop - there is a lot of overhead involved in having so many masters.

5

Hi Mhoydis,

After lowering the master to 9 from 80, new data nodes can be added to the cluster after awhile.

9 is still too many masters. You want exactly 3 masters with a quorum of 2.
Got it. I’m going to lower it to 3 now.

The problem seems related to how many concurrent connections between new node and other old instances. They establish 22~30 connections peer to peers which result in thousands connections in paralle during initilization the new node.

  • Why nodes establish such many connections to new peer? Can this be improved?
  • According to the situation while having many nodes in the cluster, does it mean that there’s scale limitaiton of a OpenSearch cluster? How’s the best practice for storing 16 billions documents per day?
# cat tcp-opensearch.txt  | grep :9300 | awk '{ print $4 }' | cut -d ":" -f1 | sort | uniq -c
      1 0.0.0.0
     24 10.42.0.104
     24 10.42.0.105
     22 10.42.0.115
     22 10.42.0.116
     22 10.42.0.117
     22 10.42.0.118
     22 10.42.0.119
     22 10.42.0.120
     22 10.42.0.121
     22 10.42.0.122
     22 10.42.0.123
     22 10.42.0.124
     24 10.42.0.125
     24 10.42.0.126
     24 10.42.0.127
     24 10.42.0.128
     24 10.42.0.129
     24 10.42.0.130
     24 10.42.0.131
     24 10.42.0.132
     24 10.42.0.133
     23 10.42.0.94
     23 10.42.0.95
     23 10.42.1.194
     24 10.42.1.204
     24 10.42.1.205
     24 10.42.1.206
     22 10.42.1.213
     22 10.42.1.215
     32 10.42.1.216
     22 10.42.1.217
     22 10.42.1.219
     32 10.42.1.222
     32 10.42.1.223
     22 10.42.1.224
     32 10.42.1.225
     22 10.42.1.226
     36 10.42.1.227
     24 10.42.1.228
     36 10.42.1.229
     24 10.42.1.230
     24 10.42.1.231
     36 10.42.1.232
     24 10.42.1.233
     24 10.42.1.234
     36 10.42.1.235
     24 10.42.1.236
     22 10.42.10.14
     22 10.42.11.19
     22 10.42.12.25
     22 10.42.13.33
     24 10.42.14.48
     32 10.42.15.27
     22 10.42.16.9
     22 10.42.17.9
     22 10.42.2.22
     24 10.42.2.24
     36 10.42.2.25
     24 10.42.2.26
     24 10.42.2.27
     32 10.42.2.28
     36 10.42.2.29
     22 10.42.2.30
     36 10.42.2.31
     24 10.42.2.32
     22 10.42.2.33
     22 10.42.2.34
     23 10.42.2.35
     22 10.42.2.37
     22 10.42.2.38
     24 10.42.2.40
     32 10.42.2.42
     24 10.42.2.43
     24 10.42.2.44
     22 10.42.2.46
     24 10.42.2.47
     22 10.42.2.50
     36 10.42.2.52
     24 10.42.2.53
     24 10.42.3.145
     23 10.42.3.179
     24 10.42.3.189
     24 10.42.3.190
     22 10.42.3.201
     22 10.42.3.202
     22 10.42.3.203
     22 10.42.3.204
     22 10.42.3.205
     32 10.42.3.206
     22 10.42.3.207
     22 10.42.3.208
     32 10.42.3.209
     22 10.42.3.210
     24 10.42.3.211
     24 10.42.3.212
     24 10.42.3.213
     36 10.42.3.214
     24 10.42.3.215
     24 10.42.3.216
     24 10.42.3.217
     24 10.42.3.218
     29 10.42.3.219
     24 10.42.3.220
     22 10.42.4.10
     22 10.42.4.11
     22 10.42.4.12
     32 10.42.4.13
     32 10.42.4.14
     36 10.42.4.16
     24 10.42.4.17
     24 10.42.4.18
     24 10.42.4.19
     24 10.42.4.20
     36 10.42.4.203
     24 10.42.4.21
     24 10.42.4.22
     24 10.42.4.23
     23 10.42.4.236
     36 10.42.4.24
     36 10.42.4.246
     24 10.42.4.247
     24 10.42.4.25
     22 10.42.4.5
     32 10.42.4.6
     22 10.42.4.7
     32 10.42.4.8
     22 10.42.4.9
     22 10.42.5.160
     24 10.42.5.163
     36 10.42.5.168
     24 10.42.5.170
     22 10.42.5.172
     32 10.42.5.173
     36 10.42.5.176
     24 10.42.5.177
     24 10.42.5.178
     24 10.42.5.179
     24 10.42.5.180
     24 10.42.5.182
     24 10.42.5.183
     22 10.42.5.184
     22 10.42.5.185
     22 10.42.5.186
     22 10.42.5.188
     32 10.42.5.189
     22 10.42.5.191
     22 10.42.5.194
     24 10.42.5.195
     24 10.42.5.196
     24 10.42.5.199
     24 10.42.5.205
     36 10.42.5.206
     23 10.42.6.191
     24 10.42.6.201
     24 10.42.6.202
     22 10.42.6.214
     22 10.42.6.215
     22 10.42.6.216
     22 10.42.6.217
     22 10.42.6.218
     22 10.42.6.219
     22 10.42.6.220
     32 10.42.6.221
     22 10.42.6.222
     22 10.42.6.223
     24 10.42.6.224
     24 10.42.6.225
     24 10.42.6.226
     24 10.42.6.227
     24 10.42.6.228
     24 10.42.6.229
     24 10.42.6.230
     24 10.42.6.231
     24 10.42.6.232
     24 10.42.6.233
     24 10.42.6.234
     32 10.42.8.10
     34 10.42.8.11
     24 10.42.8.12
     24 10.42.8.13
     24 10.42.8.14
     24 10.42.8.15
     24 10.42.8.16
     24 10.42.8.17
     36 10.42.8.18
     24 10.42.8.19
     32 10.42.8.2
     24 10.42.8.20
     24 10.42.8.21
     23 10.42.8.232
     24 10.42.8.241
     24 10.42.8.242
     36 10.42.8.243
     22 10.42.8.254
     22 10.42.8.3
     22 10.42.8.4
     22 10.42.8.5
     22 10.42.8.6
     22 10.42.8.7
     22 10.42.8.8
     22 10.42.8.9
     22 10.42.9.23
     24 10.42.9.24
root@nodexyz:~# cat tcp-opensearch.txt  | grep :9300 | awk '{ print $4 }' | cut -d ":" -f1 | sort | uniq -c | wc -l
204
6

16bil documents per day is not very much. I do that in about 15 minutes.

The problem seems related to how many concurrent connections between new node and other old instances. They establish 22~30 connections peer to peers which result in thousands connections in paralle during initilization the new node.

^ This doesn’t really make a lot of sense, I’m not sure what you’re talking about. “Thousands of connections” is not a lot. You might just need to make sure you allow enough tcp connection to your process, via the systemd unit definition.
Even if there is some case that I can’t imagine based on the description, you still should have only 3 master nodes, with a quarum of 2. If you are talking about having something external to your cluster connect to the api en masse, then use client nodes, and not master nodes, for that purpose. A “client node” is just a node with master=false and data=false.

7

Hi Mhoydis,

Ya, a few thoudsands of connections for a physical machine is not a big deal. These instances are running with k8s pod. We’ll try to tune to k8s in the host level (we do but may miss parameters).

By launching the OpenSearch cluster from Helm chart, we have dedicated master, data and ir(ingest & remote_cluster_client) roles. All documents are the output of filebeats from 1k servers for centralized logging purposes.

I think we might misunderstand the data flow. All filebeats dump logs in json to these ports. I guess the IR nodes(port:9200) receive the data streaming from Filebeats and then determine which shard and data node for storing the indexed data. The IR nodes forward the data bytes to data node(port:9300) Am I right?

metadata:
  annotations:
    field.cattle.io/publicEndpoints: '[{"addresses":["10.150.37.132"],"port":30469,"protocol":"TCP","serviceName":"cs-opensearch:opensearch-cluster-client","allNodes":true},{"addresses":["10.150.37.132"],"port":31320,"protocol":"TCP","serviceName":"cs-opensearch:opensearch-cluster-client","allNodes":true}]'

Thanks for your advice. I have a more clear picture than beginning now. Gonna need to read more documents in detail for how it works underlying. We have down the master to 3 now and waching it. There’s 100 NVMEs are going to be added to the cluster. I need to think about the best way to make this cluster more efficiently.

Hugo

8

I don’t recommend sending logs directly to opensearch from filebeat at big scale. You’ll want some kind of messaging queue, like kafka.

9

New data pod takes long time to respond. Is there any bug within the community docker image?

[opensearch@opensearch-cluster-data-0 tools]$ time ./securityadmin.sh -h 10.42.13.60  -icl   -nhnv   -cacert ~/config/root-ca.pem   -cert ~/config/kirk.pem   -key ~/config/kirk-key.pem
Security Admin v7
Will connect to 10.42.13.60:9300 ... done
ERR: Cannot connect to OpenSearch. Please refer to opensearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{ouBSFZOJQO-LMjniQVlzTw}{10.42.13.60}{10.42.13.60:9300}]]
        at org.opensearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:381)
        at org.opensearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:272)
        at org.opensearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:79)
        at org.opensearch.client.transport.TransportClient.doExecute(TransportClient.java:484)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:419)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:524)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:157)



real    0m41.568s
user    0m6.263s
sys     0m0.345s
[opensearch@opensearch-cluster-data-0 tools]$ time ./securityadmin.sh -h 10.42.13.60  -icl   -nhnv   -cacert ~/config/root-ca.pem   -cert ~/config/kirk.pem   -key ~/config/kirk-key.pem
Security Admin v7
Will connect to 10.42.13.60:9300 ... done
ERR: Cannot connect to OpenSearch. Please refer to opensearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{YCxjrb7kREOuwrUnexs5eg}{10.42.13.60}{10.42.13.60:9300}]]
        at org.opensearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:381)
        at org.opensearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:272)
        at org.opensearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:79)
        at org.opensearch.client.transport.TransportClient.doExecute(TransportClient.java:484)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:433)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:419)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:524)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:157)

real    0m38.634s
user    0m6.394s
sys     0m0.431s

The data node is not ready for minutes since it’s started.

Screen Shot 2022-04-12 at 9.14.32 PM
Screen Shot 2022-04-12 at 9.14.32 PM1864×666 144 KB

10

The docker CPU usage sticks to 100% forever. What’s consuming the cpu resource of a new spawned pod by opensearch?

CONTAINER ID   NAME                                                                                                   CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O    PIDS
61a81b48eff8   k8s_opensearch-data_opensearch-cluster-data-123_cs-opensearch_2d665deb-1221-4cbd-b7d0-788c2967f750_0   101.64%   16.88GiB / 32GiB    52.76%    0B / 0B   0B / 515MB   74
CONTAINER ID   NAME                                                                                                   CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O    PIDS
61a81b48eff8   k8s_opensearch-data_opensearch-cluster-data-123_cs-opensearch_2d665deb-1221-4cbd-b7d0-788c2967f750_0   100.73%   16.88GiB / 32GiB    52.76%    0B / 0B   0B / 515MB   74
CONTAINER ID   NAME                                                                                                   CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O    PIDS
61a81b48eff8   k8s_opensearch-data_opensearch-cluster-data-123_cs-opensearch_2d665deb-1221-4cbd-b7d0-788c2967f750_0   100.73%   16.88GiB / 32GiB    52.76%    0B / 0B   0B / 515MB   74
11

The initial issue seems related to the network issue between the two sites. The problem goes away while network issue was fixed. Just FYI.

Thanks for all the suggestions.