Versions
Opensearch 1.3.5
CentOS Linux 7 (Core)
Describe the issue:
Opensearch’s logs show an error related to SSL when this configuration should by completely disabled.
This error appears when the server is restarted. If we start the service manually, there is no problem.
Configuration:
[root@opensearch1-03 opensearch]# grep -vE '^#|^$' /etc/opensearch/opensearch.yml
cluster.name: ****_els
node.name: opensearch1-03
path.data: /var/lib/opensearch
path.logs: /var/log/opensearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["10.210.147.139", "10.210.147.140", "10.210.147.145"]
plugins.security.ssl.http.enabled: false
plugins.security.disabled: true
node.max_local_storage_nodes: 3
Relevant Logs or Screenshots:
opensearch.log
[2022-09-28T12:07:48,705][INFO ][o.o.s.a.i.AuditLogImpl ] [elasticsearch1-03] Auditing will watch <NONE> for write requests.
[2022-09-28T12:07:48,705][INFO ][o.o.s.a.i.AuditLogImpl ] [elasticsearch1-03] .opendistro_security is used as internal security index.
[2022-09-28T12:07:48,705][INFO ][o.o.s.a.i.AuditLogImpl ] [elasticsearch1-03] Internal index used for posting audit logs is null
[2022-09-28T12:07:48,706][INFO ][o.o.s.c.ConfigurationRepository] [elasticsearch1-03] Hot-reloading of audit configuration is enabled
[2022-09-28T12:07:48,706][INFO ][o.o.s.c.ConfigurationRepository] [elasticsearch1-03] Node 'elasticsearch1-03' initialized
[2022-09-28T12:08:04,619][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [elasticsearch1-03] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
...
at java.lang.Thread.run(Thread.java:829) [?:?]
[2022-09-28T12:08:04,627][INFO ][o.o.c.m.MetadataCreateIndexService] [elasticsearch1-03] [security-auditlog-2022.09.28] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2022-09-28T12:08:04,626][WARN ][o.o.h.AbstractHttpServerTransport] [elasticsearch1-03] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=null}
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
...
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
... 16 more
[2022-09-28T12:08:04,831][INFO ][o.o.c.m.MetadataMappingService] [elasticsearch1-03] [security-auditlog-2022.09.28/iZarWrkWRmaxIqoOlrYnPQ] create_mapping [_doc]
[2022-09-28T12:08:11,265][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [elasticsearch1-03] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
...
at java.lang.Thread.run(Thread.java:829) [?:?]
[2022-09-28T12:08:11,268][WARN ][o.o.h.AbstractHttpServerTransport] [elasticsearch1-03] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=null}
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
...
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.79.Final.jar:4.1.79.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
... 16 more
[2022-09-28T12:08:26,468][ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [elasticsearch1-03] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
...
at java.lang.Thread.run(Thread.java:829) [?:?]
[2022-09-28T12:08:26,469][WARN ][o.o.h.AbstractHttpServerTransport] [elasticsearch1-03] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=null}
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
...
at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a557365722d4167656e743a206375726c2f372e32392e300d0a486f73743a206c6f63616c686f73743a393230300d0a4163636570743a202a2f2a0d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1215) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) ~[netty-codec-4.1.79.Final.jar:4.1.79.Final]
... 16 more
[2022-09-28T12:08:40,226][INFO ][o.o.n.Node ] [elasticsearch1-03] stopping ...
[2022-09-28T12:08:40,226][INFO ][o.o.s.a.r.AuditMessageRouter] [elasticsearch1-03] Closing AuditMessageRouter
[2022-09-28T12:08:40,228][INFO ][o.o.s.a.s.SinkProvider ] [elasticsearch1-03] Closing InternalOpenSearchSink
[2022-09-28T12:08:40,228][INFO ][o.o.s.a.s.SinkProvider ] [elasticsearch1-03] Closing DebugSink
[2022-09-28T12:08:40,338][INFO ][o.o.n.Node ] [elasticsearch1-03] stopped
[2022-09-28T12:08:40,338][INFO ][o.o.n.Node ] [elasticsearch1-03] closing ...
[2022-09-28T12:08:40,344][INFO ][o.o.s.a.i.AuditLogImpl ] [elasticsearch1-03] Closing AuditLogImpl
[2022-09-28T12:08:40,348][INFO ][o.o.n.Node ] [elasticsearch1-03] closed
messages
Nov 28 00:13:31 opensearch1-03 systemd[1]: Starting OpenSearch...
Nov 28 00:14:47 opensearch1-03 systemd[1]: opensearch.service start operation timed out. Terminating.
Nov 28 00:14:47 opensearch1-03 systemd[1]: Failed to start OpenSearch.
Nov 28 00:14:47 opensearch1-03 systemd[1]: Unit opensearch.service entered failed state.
Nov 28 00:14:47 opensearch1-03 systemd[1]: opensearch.service failed.
