I’m working for a customers, my cluster is composed by 3 master and 9 datanode. I set a basic configuration of opensearch.yml (i disable security plugin) and the cluster work. I’ve tried a curl : curl -XGET http://localhost:9200/_cat/nodes , and i see all my nodes in the cluster.
Today the customers give e the certificated signed by the company CA. I update the configuration but when i try to do a curl using HTTPS i have this response : OpenSearch Security not initialized
I tried to run the security plugin script with the certification key and CA, but I got the following error
Security Admin v7
Will connect to localhost:9200 ... done
ERR: An unexpected ConnectionClosedException occured: Connection is closed
Trace:
org.apache.hc.core5.http.ConnectionClosedException: Connection is closed
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:1331)
at org.opensearch.client.RestClient.performRequest(RestClient.java:371)
at org.opensearch.client.RestClient.performRequest(RestClient.java:359)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:541)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:154)
Caused by: org.apache.hc.core5.http.ConnectionClosedException: Connection is closed
at org.apache.hc.core5.http2.impl.nio.AbstractH2StreamMultiplexer.onException(AbstractH2StreamMultiplexer.java:668)
at org.apache.hc.core5.http2.impl.nio.AbstractH2IOEventHandler.exception(AbstractH2IOEventHandler.java:91)
at org.apache.hc.core5.http2.impl.nio.ClientH2IOEventHandler.exception(ClientH2IOEventHandler.java:39)
at org.apache.hc.core5.reactor.ssl.SSLIOSession$1.exception(SSLIOSession.java:245)
at org.apache.hc.core5.reactor.InternalDataChannel.onException(InternalDataChannel.java:181)
at org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:55)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:176)
at org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:125)
at org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:92)
at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
at java.base/java.lang.Thread.run(Thread.java:1583)
Can you maybe check one of the master logs if you a related error?
Another thing - it might be nothing but are you using a YAML file for the OpenSearch configuration? If yes, can you please try to have the security snippet in the following format?
Thanks for the help. I update the yaml config with your format and still working as the same as mine.
This are the master log
[2025-05-20T15:30:19,270][WARN ][i.n.c.AbstractChannelHandlerContext] [opensearch-master-01] An exception 'OpenSearchSecurityException[The provided TCP channel is invalid.]; nested: DecoderException[javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown]; nested: SSLHandshakeException[Received fatal alert: certificate_unknown];' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:370) ~[?:?]
at java.base/sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?]
at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:209) ~[?:?]
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:309) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1485) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
... 16 more
[2025-05-20T15:30:19,293][INFO ][o.o.s.c.ConfigurationRepository] [opensearch-master-01] Wait for cluster to be available ...
[2025-05-20T15:44:18,121][WARN ][i.n.c.AbstractChannelHandlerContext] [opensearch-master-01] An exception 'OpenSearchSecurityException[The provided TCP channel is invalid.]; nested: DecoderException[javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication]; nested: SSLHandshakeException[Extended key usage does not permit use for TLS client authentication]; nested: ValidatorException[Extended key usage does not permit use for TLS client authentication];' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) ~[?:?]
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1263) ~[?:?]
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1167) ~[?:?]
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1144) ~[?:?]
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1691) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1537) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
... 16 more
Caused by: sun.security.validator.ValidatorException: Extended key usage does not permit use for TLS client authentication
at java.base/sun.security.validator.EndEntityChecker.checkTLSClient(EndEntityChecker.java:246) ~[?:?]
at java.base/sun.security.validator.EndEntityChecker.check(EndEntityChecker.java:151) ~[?:?]
at java.base/sun.security.validator.Validator.validate(Validator.java:269) ~[?:?]
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284) ~[?:?]
at java.base/sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138) ~[?:?]
at io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkClientTrusted(EnhancingX509ExtendedTrustManager.java:62) ~[?:?]
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1241) ~[?:?]
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1167) ~[?:?]
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1144) ~[?:?]
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1691) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1537) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]
... 16 more
I would also like to tell you that I do not have the admin certificate because I do not want the security features to be affected by the securityadmin_demo.sh
In fact I also added these parameters in the configuration to manage them autonomously from the UI
According to Configuring TLS certificates - OpenSearch Documentation you’ll need X.509 PEM certificate for plugins.security.ssl.transport.pemcert_filepath and plugins.security.ssl.http.pemcert_filepath, however as I see, you have a .cer file which is in different (DER) format.
As I understand, you can convert your DER file (.crt .cer .der) to PEM with openssl:
openssl x509 -inform der -in certificate.cer -out certificate.pem
and then configure the pem file instead. Can you please try this?
@abarocco If this is the first time you’ve started the cluster with an enabled security configuration, then you’ll need to initialize the security plugin with securityadmin.sh
Regarding the securityadmin.sh, you must use the superadmin certificate defined in the admin_dn of opensearch.yml. That certificate must have only Client Authentication EKU.
Please also be aware that in production environment you must define the list of nodes DNs to allow for secured node communication.
Hello @pablo , yes, this is the first time i started the cluster. My goal was to not use securityadmin but I will manage myself the role mapping/access ecc… so i’ve decided to put in the opensearch.yml
This is correct, but since you’re not using demo configuration, you must initilize security configuration with securityadmin.sh
Yes, you can use the same cert for both.
Certificate configuration must be kept the same across all of the nodes in the cluster. Once you enable the security plugin, all nodes will require certificates to communicate with each other.
You can’t run nodes in the mixed secured/unsecured mode.
Step 1 : Configure all the certificate on all the nodes
Step 2 : Is it right to have that logged on a single node? I expect that even if I configured only 1 node with security, at least in the logs I expected to see a different one logged
I have 1 certificate specific for any node. Example node1.pem → datanode 1 , node2.pem → datanode2 ecc…
For this settings :
plugins.security.authcz.admin_dn:
“CN=node1,OU=IS,O=IS,L=Torino,ST=Torino,C=IT”
How i configure that? I don’t have an admin certificate, can i use one of my node per the script? Or i have to create one admin certificate with alle my CN = node1, node2, node3, node4 ecc…
Step 3 : run securityadmin.sh
my concern is that the certificate I insert before launching the script does not work because even though I configured it on a master and restarted the service I get the same log:
[The provided TCP channel is invalid.]; nested: DecoderException[javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication]; nested: SSLHandshakeException[Extended key usage does not permit use for TLS client authentication]; nested: ValidatorException[Extended key usage does not permit use for TLS client authentication];' [enable DEBUG level for full stacktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Extended key usage does not permit use for TLS client authentication
I think you can create a self-signed admin certificate but I’m not sure if it will work with your other certs. As far as I understand, all certificates has to be signed by the same CA.
See:
Also as I saw, it is not supported to use a node certificate for admin certificate. This actually worked for me with OpenSearch 2.18 but it gave an ERROR that it is not supported anymore.
@abarocco as far as I understand, you can configure whatever subject you want in your admin certificate. Just it has to match with what you configure for plugins.security.authcz.admin_dn in your opensearch.yml file.
From the doc i’ve seen that if i use another CA i have to concatenate both CA and put it in pemtrustedcas_filepath
I restart the cluster but same error
[2025-05-22T16:34:02,578][ERROR][o.o.s.t.SecurityRequestHandler] [opensearch-master-01] OpenSearchException[Node presenting certificate with SSL Principal {CN=vmgclalte1748.syssede.systest.sanpaoloimi.com,OU=Intesa Sanpaolo S.p.A.,O=Intesa Sanpaolo S.p.A.,L=Torino,ST=Torino,C=IT} could not securely connect to the cluster. Please ensure the principal is correct and present in the nodes_dn list.]