Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

sharath · November 24, 2021, 12:06pm

We getting below error in Kubernetes

Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
… 16 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]
… 16 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certif

Configuration

config:
#opendistro_security.disabled: true
opendistro_security.ssl.transport.pemcert_filepath: elk-transport-crt.pem
opendistro_security.ssl.transport.pemkey_filepath: elk-transport-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: elk-transport-root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false

opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: elk-rest-crt.pem
opendistro_security.ssl.http.pemkey_filepath: elk-rest-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: elk-rest-root-ca.pem

opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
- 'CN=admin,OU=IT,O=xxx,L=PA,ST=CA,C=US'
opendistro_security.nodes_dn:
- 'CN=*,OU=IT,O=xxx,L=PA,ST=CA,C=US'
- 'CN=od-k8s*'

opendistro_security.advanced_modules_enabled: true
opendistro_security.roles_mapping_resolution: BOTH
opendistro_security.audit.ignore_users: ['kibanaserver']
opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
## Example Config
opendistro_security.allow_unsafe_democertificates: false

We are using below url to generate certificate

https://opendistro.github.io/for-elasticsearch-docs/old/0.9.0/docs/security/generate-certificates/

DrEdWilliams · November 24, 2021, 8:45pm

Generally, this message means that your certs/keys don’t line up with your CAs.

It’s not clear from what you provided what connection is triggering this message – unfortunately, the error message is generic to ANY SSL connection … more context would be useful.

You have observed that there are two separate sets of certs/keys/CAs … you need to make sure your client cert is aligning with the right CA – generally this will be the REST CA … but again, more context as to what is trying to make the connection would be useful.

Gsmitt · November 25, 2021, 4:59am

Hello,

Try setting the full path where your certificates are. For testing, I used Elasticsearch directory since elastic owns that directory.

Example:

opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: /etc/elasticsearch/admin.pem
opendistro_security.ssl.http.pemkey_filepath: /etc/elasticsearch/admin-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: /etc/elasticsearch/root-ca.pem

Hope that helps

sharath · November 25, 2021, 5:23am

Hi Williams,

We have followed the document provided by opendistro. Any suggest on this

Thanks

sharath · November 25, 2021, 5:24am

Hi Smit,

I used full path, still same error

Thanks
Sharath

Gsmitt · November 25, 2021, 5:27am

Just a guess, can Elastic access you keystore and certs? I used JAVA so my keystore is named cacerts. Other then that I not sure.

EDIT: When you used the full path in your file did you execute your ./securityadmin.sh?

DrEdWilliams · November 25, 2021, 4:36pm

@sharath - Let’s try to narrow this down a bit …

How many nodes are in your cluster?
Are they all the same (master/ingest/data) or have you separated the functions?
At what point do the errors show up – as the cluster is coming up, or when you try to access it with kibana?
Have you been able to run the securityadmin.sh script to load the security configuration?

What response do you get a response to a cluster query using curl (filling in your master node name):

curl -k https://master-node:9200

A quick way to check your certs vs CAs is to use openssl:

openssl verify -verbose -CAfile <ca cert file> <certificate file>

That may show some issues. @Gsmitt has a good point – permissions on the certificate files must be set so that the user running elasticsearch has at least read access.

It may also help to see the log entries as the cluster is coming up, not just the errors.

I know this is a lot of questions … but we need to identify the point at which the failure occurs.

elham · July 27, 2022, 9:48am

Hi @DrEdWilliams

i get this for curl command:

and when i run securityadmin.sh command i face this error:


PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Gsmitt · July 28, 2022, 4:12am

@elham

Whenever Java attempts to connect to another application over SSL it will only be able to connect to that application if it can trust it. The way trust is handled in the Java world is that you have a keystore (typically $JAVA_HOME/lib/security/cacerts), also known as the truststore.

Just a guess, either wrong certificate in in the keystore or you need to set the path in the application using this Keystore or the certificates maybe incorrect.

ogulman · November 28, 2023, 2:06pm

Hi,
I do have the same issue (OpenSearch 2.11):

sh-5.2$ /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cacert /usr/share/opensearch/config/admin-root-ca.pem -cert /usr/share/opensearch/config/admin.pem -key /usr/share/opensearch/config/admin.key.pem -cd /usr/share/opensearch/config/opensearch-security/
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to localhost:9200 ... done
ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
Trace:
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting.
        at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:948)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:333)
        at org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:573)
        at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:163)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:378)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:316)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264)
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:289)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:357)
        at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:545)
        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
        at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
        ... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
        at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:127)
        at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
        at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
        ... 24 more

I’ve generate admin certs using this manual:

Never had it with open distro.

ogulman · November 28, 2023, 2:20pm

openssl verify -verbose -CAfile root-ca.pem admin.pem
Certificate request self-signature ok

siradjedd · April 16, 2024, 3:46pm

@ogulman any updates ?

ogulman · April 16, 2024, 4:01pm

That was something on my side.

siradjedd · April 16, 2024, 7:16pm

@ogulman what was the problem ? i have the same issue when i try to connect java app to opensearch on k8s

Topic		Replies	Views
ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: Security	9	2317	April 10, 2023
Securityadmin.sh unable to find valid certification path to requested target error Security configure , security-issue	2	762	November 1, 2023
"SSL Problem PKIX path building failed" & "certificate_unknown" when changing admin certificates Security	1	5064	December 18, 2019
Error while executing securityadmin.sh in OS 1.2.4 Security security-issue	19	826	October 26, 2023
Exception publishing Message PKIX path building failed Error Alerting	4	1215	October 17, 2020

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Related topics