# Logstash Opensearch connection keeps failing **URL:** \ **Category:** OpenSearch\ **Tags:** troubleshoot\ **Created:** [December 5, 2023, 2:21pm UTC](https://forum.opensearch.org/t/logstash-opensearch-connection-keeps-failing/16957 "2023-12-05T14:21:35Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![georg](https://avatars.discourse-cdn.com/v4/letter/g/9f8e36/32.png) [@georg](https://forum.opensearch.org/u/georg)\ **Post date:** [December 5, 2023, 2:21pm UTC](https://forum.opensearch.org/t/logstash-opensearch-connection-keeps-failing/16957/1 "2023-12-05T14:21:35Z")
I’m using: **Opensearch 2.11** **Logstash 8.9** My setup has two Logstash Server with equal Config and several Pipelines. My Opensearch cluster has two data/ingestion nodes and three master nodes. The connection from the Logstash servers to one of the data/ingestion nodes keep failing (the other works fine). The connection keeps restoring and as far as I can tell there is no data loss. This keeps happening several times per minute. The opensearch.service also fails from time to time (every other day). All instances run on Linux machines. I am trying to debug this problem for some days now and have tried performance tuning, but to no success. Although the cluster has to ingest a high amount of events, I don’t think, that this is the problem, since we have a similar setup with no problems and only one of two data/ingestion nodes keeps failing. **Configuration:** **Opensearch:** > cluster.name: XXX-cluster > node.name: data01 > node.roles: [data,ingest] > path.data: /var/lib/opensearch > path.logs: /var/log/opensearch > bootstrap.memory\_lock: true > network.host: XX.XX.XX.XX > http.port: 9200 > discovery.seed\_hosts: [“master01.xx.xx”, “master02.xx.xx”, “master03.xx.xx”, “data01.xx.xx”, “data02.xx.xx”] > cluster.initial\_cluster\_manager\_nodes: [“master01.xx.xx”, “master02.xx.xx”, “master03.xx.xx”] > plugins.security.ssl.transport.pemcert\_filepath: /etc/opensearch/certs/data01.crt > plugins.security.ssl.transport.pemkey\_filepath: /etc/opensearch/certs/data01.pem > plugins.security.ssl.transport.pemtrustedcas\_filepath: /etc/opensearch/certs/ca.pem > plugins.security.ssl.transport.enforce\_hostname\_verification: false > plugins.security.ssl.http.enabled: true > plugins.security.ssl.http.pemcert\_filepath: /etc/opensearch/certs/data01.crt > plugins.security.ssl.http.pemkey\_filepath: /etc/opensearch/certs/data01.pem > plugins.security.ssl.http.pemtrustedcas\_filepath: /etc/opensearch/certs/ca.pem > plugins.security.allow\_unsafe\_democertificates: false > plugins.security.allow\_default\_init\_securityindex: true > plugins.security.authcz.admin\_dn: > > - ‘XXX’ > plugins.security.nodes\_dn: > - ‘XXX’ > - … > - … > plugins.security.audit.type: internal\_opensearch > plugins.security.enable\_snapshot\_restore\_privilege: true > plugins.security.check\_snapshot\_restore\_write\_privileges: true > plugins.security.restapi.roles\_enabled: [“all\_access”, “security\_rest\_api\_access”] > plugins.security.system\_indices.enabled: true > plugins.security.system\_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert\*”, “.opendistro-anomaly-results\*”, “.opendistro-anomaly-detector\*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-_", ".opensearch-notifications-_”, “.opensearch-notebooks”, “.opensearch-observability”, “.ql-datasources”, “.opendistro-asynchronous-search-response\*”, “.replication-metadata-store”, “.opensearch-knn-models”] > > http.pipelining.max\_events: 100000 > knn.algo\_param.index\_thread\_qty: 20 **Logstash-Pipeline:** > output{ > opensearch { > hosts =\> [“ht…data01.xxx.xxx:9200”,“ht…data02.xxx.xxx:9200”] > user =\> “XXX” > password =\> “XXX” > index =\> “web-%{+yyyy.MM.dd}” > ssl =\> true > ssl\_certificate\_verification =\> true > cacert =\> “/etc/logstash/certs/ca.pem” > timeout =\> 600 > } > } \*\*Relevant Logs: \*\* **Opensearch (shortened for readability):** > [2023-12-05T15:02:47,424][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [data01.xx.xx] Exception during establishing a SSL connection: null > [2023-12-05T15:02:47,424][WARN][o.o.h.AbstractHttpServerTransport] [data01.xx.xx] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/xx.xx.xx.xx:9200, remoteAddress=/xx.xx.xx.xx:59962} > io.netty.handler.codec.compression.DecompressionException: CRC value mismatch. Expected: 1398561884, Got: 1034882459 > at java.lang.Thread.run(Thread.java:833) [?:?] > [2023-12-05T15:02:47,426][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [data01.xx.xx] Exception during establishing a SSL connection: io.netty.handler.codec.PrematureChannelClosureException: Channel closed while still aggregating message > io.netty.handler.codec.PrematureChannelClosureException: Channel closed while still aggregating message > at java.lang.Thread.run(Thread.java:833) [?:?] > [2023-12-05T15:02:47,426][WARN][o.o.h.AbstractHttpServerTransport] [data01.xx.xx] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/xx.xx.xx.xx:9200, remoteAddress=/xx.xx.xx.xx:59962} > io.netty.handler.codec.PrematureChannelClosureException: Channel closed while still aggregating message > at java.lang.Thread.run(Thread.java:833) [?:?] **Logstash:** > [2023-12-05T15:05:56,156][WARN][logstash.outputs.opensearch][web] Restored connection to OpenSearch instance {:url=\>“ht…logstash\_user:xxxxxx@data01.xx.xx:9200/”} > [2023-12-05T15:07:19,747][WARN][logstash.outputs.opensearch][web][2964b1369773f13ee2024ed0ffaddf9cf07364df9e7737ea71148f2051d37781] Marking url as dead. Last error: [LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError] OpenSearch Unreachable: [ht…logstash\_user:xxxxxx@data01.xx.xx:9200/][Manticore::ClientProtocolException] data01.xx.xx:9200 failed to respond {:url=\>ht…logstash\_user:xxxxxx@data01.xx.xx:9200/, :error\_message=\>“OpenSearch Unreachable: [ht…logstash\_user:xxxxxx@data01.xx.xx:9200/][Manticore::ClientProtocolException] data01.xx.xx:9200 failed to respond”, :error\_class=\>“LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError”} > [2023-12-05T15:07:19,747][ERROR][logstash.outputs.opensearch][web][2964b1369773f13ee2024ed0ffaddf9cf07364df9e7737ea71148f2051d37781] Attempted to send a bulk request but OpenSearch appears to be unreachable or down {:message=\>“OpenSearch Unreachable: [ht…logstash\_user:xxxxxx@data01xx.xx:9200/][Manticore::ClientProtocolException] data01.xx.xx:9200 failed to respond”, :exception=\>LogStash::Outputs::OpenSearch::HttpClient::Pool::HostUnreachableError, :will\_retry\_in\_seconds=\>2} > [2023-12-05T15:07:21,204][WARN][logstash.outputs.opensearch][web] Restored connection to OpenSearch instance {:url=\>“ht…logstash\_user:xxxxxx@data01.xx.xx:9200/”} I’m grateful for any help since I’m quite lost as to how to solve this problem… ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [December 8, 2023, 1:39am UTC](https://forum.opensearch.org/t/logstash-opensearch-connection-keeps-failing/16957/2 "2023-12-08T01:39:45Z")
> [@georg](#): > > cacert =\> “/etc/logstash/certs/ca.pem” Is that Logstash’s or OpenSearch’s CA cert? ---
**Author:** ![georg](https://avatars.discourse-cdn.com/v4/letter/g/9f8e36/32.png) [@georg](https://forum.opensearch.org/u/georg)\ **Post date:** [December 11, 2023, 10:54am UTC](https://forum.opensearch.org/t/logstash-opensearch-connection-keeps-failing/16957/3 "2023-12-11T10:54:56Z")
Thank’s for the reply! This is the root certificate to establish trust to the Opensearch cluster. It seems though there was a bug with Opensearch. Since the update from Opensearch 2.11.0 to 2.11.1 the connection resets only once or twice per hour, which seems manageable to me. Although I’m stilled puzzled why this keeps happening, this isn’t a major problem to our system anymore. ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [December 12, 2023, 10:13am UTC](https://forum.opensearch.org/t/logstash-opensearch-connection-keeps-failing/16957/4 "2023-12-12T10:13:11Z")
@georg HostUnreachableError would be related to connection issues. How Logstash connect to OpenSearch nodes? Do you use any reverse proxy or a load balancer in the front of the cluster? Do you use VPN connection between Logstash and OpenSearch nodes? How far, physically, is the Logstash from OpenSearch ingest nodes? How did you deploy the cluster? > [@georg](#): > > The opensearch.service also fails from time to time What do you mean by that? Is the OpenSearch service unstable or are you referring to the reported disconnect errors in OpenSearch logs? ---
**Author:** ![taltsafrir](https://avatars.discourse-cdn.com/v4/letter/t/a698b9/32.png) [@taltsafrir](https://forum.opensearch.org/u/taltsafrir)\ **Post date:** [August 16, 2024, 8:52pm UTC](https://forum.opensearch.org/t/logstash-opensearch-connection-keeps-failing/16957/5 "2024-08-16T20:52:29Z")
I’m getting the same errors with upgrading from 2.7 to 2.16 ---
**Author:** ![taltsafrir](https://avatars.discourse-cdn.com/v4/letter/t/a698b9/32.png) [@taltsafrir](https://forum.opensearch.org/u/taltsafrir)\ **Post date:** [August 22, 2024, 6:38am UTC](https://forum.opensearch.org/t/logstash-opensearch-connection-keeps-failing/16957/6 "2024-08-22T06:38:19Z")
@pablo do we have any solution? ---
**Author:** ![pablo](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/pablo/32/10363_2.png) [@pablo](https://forum.opensearch.org/u/pablo)\ **Post date:** [August 22, 2024, 11:01am UTC](https://forum.opensearch.org/t/logstash-opensearch-connection-keeps-failing/16957/7 "2024-08-22T11:01:43Z")
@taltsafrir Could you open a new thread and share your logstash config file, config.yml and opensearch.yml files?