Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Runs on an ARM AWS Server
Amazon Linux 2
OpenSearch version 2.8.0
Security plugin version 2.8.0
Describe the issue :
We are trying to set secure connections, using certs.
We followed documentation without success, knowing we have 1 key difference :
- We use a certificate signed by Gandi (Gandi.net - Gandi.net: Domain Names, Web Hosting, SSL Certificates and Emails) for the
httplayer. - We use a self signed certificate for the
transportlayer.
As expected, we declare a chained cert for the http layer.
querying the opensearch server on port 9200 gives us a clean HTTPS connection, with valid cert, no problem with that.
Using opendistro-security.sh will always fail we an error we do understand, but we don’t understand why it triggers as to us everything is clear:
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755 **
**************************************************************************
Security Admin v7
Will connect to OURDOMAINNAME:9200 ... done
ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:947)
at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:465)
at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:162)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:288)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:356)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:547)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 24 more
Configuration:
Stripped configuration to keep most interesting parts:
plugins.security.ssl.transport.pemcert_filepath: config/node1.pem
plugins.security.ssl.transport.pemkey_filepath: config/node1-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: config/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: config/OURDOMAINNAME-chained.pem
plugins.security.ssl.http.pemkey_filepath: config/OURDOMAINNAME-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: config/USERTrustRSAAddTrustCA.pem
So as you can see, for transport we do have self-signed cert, created with root-ca.pem.
For http, we have cat Gandi’s cert with our cert, and declared inside pemtrustedcas_filepath the cert of the root CA that signs Gandi’s cert.
We can validate the cert using both curl/openssl, there is no complains about our cert chain.
But as soon as we fire opendistro-security.sh, it gets ugly, as per mentioned above.
We also tried plain java client, to see if it would be impacted by a cert issue:
/usr/share/opensearch/jdk/bin/java SSLPoke OURDOMAINNAME 9200
But it works perfectly.
The command we use is :
JAVA_HOME="/usr/share/opensearch/jdk" /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cd "/etc/opensearch/opensearch-security/" -icl -key "/etc/opensearch/config/admin-key.pem" -cert "/etc/opensearch/config/admin.pem" -cacert /etc/opensearch/config/root-ca.pem -nhnv -h "OURDOMAINNAME"
Before you ask, i confirm that our admin cert is made from root-ca’s cert, just like node1’s cert.
As an additional information, our process is working fine under opendistro 1.13.2, but we face this issue under opensearch 2.8.0
Worth noting:
Using only our self signed cert for both http & transport solves the issue with securityadmin.sh but this is not what we would like to have.
Using only our gandi cert for both http & transport does not solves the issue.