Unable to update TLS certificates

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.5

Describe the issue:
We were using the generated demo-certificates. Now we plan on migrating the certificates managed by cert-manager and let’s encrypt. (Running the cluster on OpenSearch operator). If I try to change the certificate from generate: true to fetch from a secret, or vice versa, the rollout fails with PKIX path building failed: unable to find valid certification path to requested target. however, If I delete the CR first and create the same cluster from scratch it works fine.

OpenSearch dashboard is deployed with ingress at opensearch.mycompany.com
OpenSearch Server: api.opensearch.mycompany.com
admin certificate: admin.opensearch.mycompany.com
http certificate: http.opensearch.mycompany.com
transport: transport.opensearch.mycompany.com

While the generated certificates followed something like
CN=admin, OU=opensearch-cluster-name
CN=node, OU=opensearch-cluster-name

I tried using certificates provisioned by cert-manager (http/transport). as well as the certificates created by ingress controller. Getting the same error. What am I doing wrong here? Is it possible to change the certificates without recreating the whole cluster?

Configuration:

"plugins.security.ssl.transport.enforce_hostname_verification": false
"plugins.security.ssl.transport.pemkey_filepath": "lets-encrypt/tls.key"
"plugins.security.ssl.transport.pemcert_filepath": "lets-encrypt/tls.crt"
"plugins.security.ssl.transport.pemtrustedcas_filepath": "lets-encrypt/tls.crt"
"plugins.security.ssl.http.pemkey_filepath": "lets-encrypt/tls.key"
"plugins.security.ssl.http.pemcert_filepath": "lets-encrypt/tls.crt"
"plugins.security.ssl.http.pemtrustedcas_filepath": "lets-encrypt/tls.crt"

Relevant Logs or Screenshots:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
    at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:947)
    at org.opensearch.client.RestClient.performRequest(RestClient.java:332)
    at org.opensearch.client.RestClient.performRequest(RestClient.java:320)
    at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:462)
    at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:159)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:371)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
    at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:309)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:654)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
    at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
    at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
    at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
    at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:288)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:356)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:547)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
    at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
    at java.base/sun.security.validator.Validator.validate(Validator.java:264)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
    at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
    at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
    ... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
    at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
    at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
    at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
    ... 24 more

Running the securityadmin.sh returns the same error as above on node restarts.

2

In my config I put the full path to the certs and all worked.
ie:

plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/root-ca.pem
1 Like
3

Hi @maulin.shah,

I noticed that you’re using the same file for both the pemcert_filepath and pemtrustedcas_filepath parameters.

According to the documentation, these are meant to be different files. The pemcert_filepath should contain the node certificate, while the pemtrustedcas_filepath should contain the rootCA certificate.

You can concatenate the node and intermediate certificates in pemcert, but then you should use only the Root CA in pemtrustedcas.

4

Hi @jasonrojas / @Eugene7

Apologies for the late response.
I tried using full path, as wel as supplying the CA cert. unfortunately, I am still getting the same error.

Furthermore, I tried

  • Upgrading to OpenSearch 2.13.0,
  • Concatenating intermediate certificate to the CA cert and keeping only node cert to pem_file: Opensearch security plugin with certificate chain - #2 by pablo
  • Using self-signed certificates
  • Importing the Root CA certificate to Java truststore with keytool inside of the pod
  • Using ISRG Root X1 and Amazon Root CA 1.

What’s odd is that if I use the same configuration and same certificates to spin up a new OpenSearchCluster, it works. However, when updating the certificate, I get this error.

I would have tried to use the PKCS#12/JKS instead of PKCS#8, but the secrets issued by cert manager do not include a truststore.

5

@maulin.shah one other thing worth trying is ensuring that you have hot reloading of certificates enabled, and then using the REST API to actually cause the reload. Docs here: https://opensearch.org/docs/latest/security/configuration/tls/#hot-reloading-tls-certificates