Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch Version Used: 2.12.0
Describe the issue:
We are deploying OpenSearch in a Kubernetes environment with security enabled, configuring the following cipher:
plugins.security.ssl.http.enabled_ciphers:
- “TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384”
When the opensearch is deployed, security admin is failing with below error
------------------------------------------------------------
Output of running security admin -
securityadmin started
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755 **
**************************************************************************
Security Admin v7
Will connect to va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg:9200 ... done
ERR: An unexpected SSLHandshakeException occured: Received fatal alert: handshake_failure
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:948)
at org.opensearch.client.RestClient.performRequest(RestClient.java:333)
at org.opensearch.client.RestClient.performRequest(RestClient.java:321)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:572)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:162)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.decode(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:279)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:333)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:545)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Unknown Source)
securityadmin ended
------------------------------------------------------------
Securityadmin failed due to above error
In the OpenSearch logs, we observe the following cipher set:
logger":"o.o.s.s.DefaultSecurityKeyStore","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg] ","log":{"message":"JVM supports the following 49 ciphers [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]"}}
{"type":"log","host":"va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg.varun","container":"is-client","level":"DEBUG","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-18T14:06:50.618Z","logger":"i.n.h.s.JdkSslContext","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg] ","log":{"message":"Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384]"}}
{"type":"log","host":"va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg.varun","container":"is-client","level":"DEBUG","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-18T14:06:51.719Z","logger":"o.o.s.s.DefaultSecurityKeyStore","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg] ","log":{"message":"sslTransportClientProvider:JDK with ciphers [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA]"}}
{"type":"log","host":"va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg.varun","container":"is-client","level":"DEBUG","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-18T14:06:51.719Z","logger":"o.o.s.s.DefaultSecurityKeyStore","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg] ","log":{"message":"sslTransportServerProvider:JDK with ciphers [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA]"}}
The configured cipher is present in the opensearch logs. Please refer below screenshot for reference.
The configured cipher is present in OpenSearch. However, the security administration process fails, and the logs in opensearch indicate a “cipher not found” issue
{"type":"log","host":"va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg.varun","container":"is-client","level":"DEBUG","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-18T14:16:24.901Z","logger":"o.o.n.r.t.AverageMemoryUsageTracker","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg] ","log":{"message":"Recording memory usage: 66%"}}
{"type":"log","host":"va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg.varun","container":"is-client","level":"ERROR","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-18T14:16:25.016Z","logger":"o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg] ","log":{"message":"Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: no cipher suites in common"}}
javax.net.ssl.SSLHandshakeException: no cipher suites in common
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.ServerHello$T12ServerHelloProducer.chooseCipherSuite(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.ServerHello$T12ServerHelloProducer.produce(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(Native Method) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1651) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1497) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.106.Final.jar:4.1.106.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.106.Final.jar:4.1.106.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Could you please provide insights into why the security admin process is failing despite configuring ciphers available in the JVM?