Opensearch secuirty admin process is failing , when opensearch is deployed by configuring available ciphers

Varun_Srinivasa · April 19, 2024, 4:46am

Versions

OpenSearch Version Used: 2.12.0

Describe the issue:

We are deploying OpenSearch in a Kubernetes environment with security enabled, configuring the following cipher:

plugins.security.ssl.http.enabled_ciphers:
   - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"

Upon deployment, the security admin process is encountering the following error:

------------------------------------------------------------
Output of running security admin -
 securityadmin started
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg:9200 ... done
ERR: An unexpected SSLHandshakeException occured: Received fatal alert: handshake_failure
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
        at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:948)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:333)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:572)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:162)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
        at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Unknown Source)
        at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
        at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl.decode(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(Unknown Source)
        at java.base/javax.net.ssl.SSLEngine.unwrap(Unknown Source)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:279)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:333)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:545)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.base/java.lang.Thread.run(Unknown Source)


securityadmin ended
------------------------------------------------------------
Securityadmin failed due to above error

In the OpenSearch logs, we observe the following cipher set:

{"type":"log","host":"va-ifd-bssc-indexsearch-client-895f9bcb8-22xpd.va-test","container":"is-client","level":"DEBUG","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-17T06:55:08.715Z","logger":"o.o.s.s.DefaultSecurityKeyStore","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-895f9bcb8-22xpd] ","log":{"message":"JVM supports the following 49 ciphers [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]"}}
 
{"type":"log","host":"va-ifd-bssc-indexsearch-client-895f9bcb8-22xpd.va-test","container":"is-client","level":"DEBUG","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-17T06:55:08.909Z","logger":"i.n.h.s.JdkSslContext","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-895f9bcb8-22xpd] ","log":{"message":"Default cipher suites (JDK): [TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384]"}}
 
{"type":"log","host":"va-ifd-bssc-indexsearch-client-895f9bcb8-22xpd.va-test","container":"is-client","level":"DEBUG","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-17T06:55:10.016Z","logger":"o.o.s.s.DefaultSecurityKeyStore","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-895f9bcb8-22xpd] ","log":{"message":"sslTransportClientProvider:JDK with ciphers [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA]"}}
 
{"type":"log","host":"va-ifd-bssc-indexsearch-client-895f9bcb8-22xpd.va-test","container":"is-client","level":"DEBUG","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-17T06:55:10.016Z","logger":"o.o.s.s.DefaultSecurityKeyStore","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-895f9bcb8-22xpd] ","log":{"message":"sslTransportServerProvider:JDK with ciphers [TLS_AES_256_GCM_SHA384, TLS_AES_128_GCM_SHA256, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA]"}}

The configured cipher is present in OpenSearch logs. Please refer below screenshot for reference.

The configured cipher is present in OpenSearch. However, the security admin process fails, and opensearch the logs indicate a “cipher not found” issue. Please refer below log for reference.

{"type":"log","host":"va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg.varun","container":"is-client","level":"ERROR","systemid":"BSSC-1234","system":"BSSC","time": "2024-04-18T14:18:45.016Z","logger":"o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport","timezone":"UTC","marker":"[va-ifd-bssc-indexsearch-client-65c5cbff5c-cjvvg] ","log":{"message":"Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: no cipher suites in common"}}
javax.net.ssl.SSLHandshakeException: no cipher suites in common
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.ServerHello$T12ServerHelloProducer.chooseCipherSuite(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.ServerHello$T12ServerHelloProducer.produce(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.SSLHandshake.produce(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.ClientHello$ClientHelloConsumer.consume(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(Unknown Source) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(Native Method) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(Unknown Source) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1651) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1497) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1338) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1387) ~[netty-handler-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.106.Final.jar:4.1.106.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.106.Final.jar:4.1.106.Final]
        at java.base/java.lang.Thread.run(Unknown Source) [?:?]

Could you please provide insights into why the security admin process is failing despite configuring ciphers available in the JVM?

Gsmitt · April 20, 2024, 2:51am

Hey @Varun_Srinivasa

Did a quick look over and I found this in your logs

javax.net.ssl.SSLHandshakeException: no cipher suites in common

It seams the certificates your using in OS/OSD configuration files, does match up with your ciphers.

By chance does it work without enabling/configuring plugins.security.ssl.http.enabled_ciphers?

Varun_Srinivasa · April 22, 2024, 6:06am

Hi @Gsmitt ,

Thanks for getting back to us.

We’ve noticed that when we don’t configure the property plugins.security.ssl.http.enabled_ciphers, we don’t encounter any issues.

It seems that there might be a mismatch between the certificates you’re using in the OS/OSD configuration files and your ciphers–>Could you please explain how you confirmed that the certificates configured in OpenSearch are matching with the ciphers?

Regards,
Varun S

Gsmitt · April 23, 2024, 12:15am

Hey<

You can grep it.

echo -n '' | openssl s_client -connect opensearch.mydomain.com:443 2>&1 | grep -Po "(?<=Cipher is ).*$"

Example:

Topic		Replies	Views
The security admin process in OpenSearch fails when deployed with configured ciphers OpenSearch install , security-issue	1	210	April 23, 2024
SSL HandShake issues using self signed admin-cert and http letsencrypt certificates Security	1	93	August 12, 2024
Securityadmin.sh not working due to admin certificate issue Security	4	36	November 5, 2024
Securityadmin.sh errors when connecting opensearch through http Security troubleshoot	4	3692	June 2, 2022
Query: To run securityadmin.sh when TLS is disabled on Rest layer Security	1	656	July 18, 2022

Opensearch secuirty admin process is failing , when opensearch is deployed by configuring available ciphers

Related topics