TLS/SSL issue - javax.net.ssl.SSLHandshakeException

socket370 · November 16, 2020, 11:47pm

I am getting the forrlow ERRORs/WARNINGs in my logs:

2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,473][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49972, remoteAddress=DATANODE04_P/246.802.468.187:9300}], closing connection
[2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,473][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:47688, remoteAddress=DATANODE05_P/246.802.468.188:9300}], closing connection
[2020-11-13T10:20:49,474][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49254, remoteAddress=DATANODE02_P/246.802.468.185:9300}], closing connection
[2020-11-13T10:20:49,479][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:60736, remoteAddress=DATANODE03_P/246.802.468.186:9300}], closing connection
[2020-11-13T10:20:49,465][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:49,482][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:43392, remoteAddress=DATANODE01_P/246.802.468.184:9300}], closing connection
[2020-11-13T10:20:49,501][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[2020-11-13T10:20:49,503][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:9300, remoteAddress=/246.802.468.188:43492}], closing connection
[2020-11-13T10:20:49,509][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[2020-11-13T10:20:49,513][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:9300, remoteAddress=/246.802.468.184:40572}], closing connection
[2020-11-13T10:20:50,317][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,318][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,319][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:60744, remoteAddress=DATANODE03_P/246.802.468.186:9300}], closing connection
[2020-11-13T10:20:50,321][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,322][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,317][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:50,320][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:47698, remoteAddress=DATANODE05_P/246.802.468.188:9300}], closing connection
[2020-11-13T10:20:50,323][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:43408, remoteAddress=DATANODE01_P/246.802.468.184:9300}], closing connection
[2020-11-13T10:20:50,324][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49266, remoteAddress=DATANODE02_P/246.802.468.185:9300}], closing connection
[2020-11-13T10:20:50,324][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49976, remoteAddress=DATANODE04_P/246.802.468.187:9300}], closing connection
[2020-11-13T10:20:50,493][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[2020-11-13T10:20:50,494][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:9300, remoteAddress=/246.802.468.188:43508}], closing connection
[2020-11-13T10:20:50,500][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[2020-11-13T10:20:50,501][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:9300, remoteAddress=/246.802.468.184:40582}], closing connection
[2020-11-13T10:20:51,324][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:51,326][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:51,326][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:51,328][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:49270, remoteAddress=DATANODE02_P/246.802.468.185:9300}], closing connection
[2020-11-13T10:20:51,328][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [MASTERNODE01_P] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
[2020-11-13T10:20:51,329][WARN ][o.e.t.TcpTransport ] [MASTERNODE01_P] exception caught on transport layer [Netty4TcpChannel{localAddress=/246.802.468.181:60762, remoteAddress=DATANODE03_P/246.802.468.186:9300}], closing connection

Here is the bottom portion of my elasticsearch.yml:

WARNING: revise all the lines below before you go into production

opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.transport.resolve_hostname: false
opendistro_security.ssl.transport.pemcert_filepath: master001.pem
opendistro_security.ssl.transport.pemkey_filepath: master001.key
opendistro_security.ssl.transport.pemtrustedcas_filepath: ca-chain-bundle.pem
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: master001.pem
opendistro_security.ssl.http.pemkey_filepath: master001.key
opendistro_security.ssl.http.pemtrustedcas_filepath: ca-chain-bundle.pem
opendistro_security.allow_unsafe_democertificates: false
opendistro_security.allow_default_init_securityindex: true
opendistro_security.ssl.http.enabled_protocols:

“TLSv1.2”
“TLSv1.3”
opendistro_security.ssl.http.enabled_ciphers:
“TLS_AES_256_GCM_SHA384”
“TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”
“TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384”
“TLS_DHE_RSA_WITH_AES_256_GCM_SHA384”
“TLS_DHE_RSA_WITH_AES_128_GCM_SHA256”
opendistro_security.authcz.admin_dn:
‘O=SomeOrg1,O=SomeOrg2,L=SomeCity,S=SomeState,C=SomeCountry’
opendistro_security.nodes_dn:
'CN= NODE ’
opendistro_security.ssl.transport.enabled_protocols:
“TLSv1.2”
“TLSv1.3”
opendistro_security.ssl.transport.enabled_ciphers:
“TLS_AES_256_GCM_SHA384”
“TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256”
“TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384”
“TLS_DHE_RSA_WITH_AES_256_GCM_SHA384”
“TLS_DHE_RSA_WITH_AES_128_GCM_SHA256”
opendistro_security.ssl.http.clientauth_mode: OPTIONAL
cluster.routing.allocation.disk.threshold_enabled: true
opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
opendistro_security.system_indices.enabled: true
opendistro_security.system_indices.indices: [“.opendistro-alerting-config”, “.opendistro-alerting-alert*”]

The master01.pem and master.key were create via a csr to my corporate CA – I am not using an internal elasticsearch CA.

Seems to suggest that this is a keystore issue. However, my understanding per

Is that using a keystore is optional. I have my RootCA and my IssuingCA certs concatenated in ca-chain-bundle.pem

socket370 · November 17, 2020, 12:19pm

I created keystore using these directions:

https://docs.oracle.com/en/database/other-databases/nosql-database/12.2.4.5/security/import-key-pair-java-keystore.html

and the bottom of this link to create the truststore:

https://docs.oracle.com/en/database/other-databases/nosql-database/12.2.4.5/security/java-keystore-preparation.html

This seems to have solved the issue.

socket370 · November 17, 2020, 3:58pm

I am still seeing errors:

Exception during establishing a SSL connection: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

socket370 · November 17, 2020, 9:32pm

Okay for ODFE version 1.11 - the keystore / truststore instruction I posted previously work. However, I found through another issue I posted about regarding another issue - you need to add this to this config file:

opendistro_security.ssl.http.keystore_keypassword

And whatever the password that was created.

socket370 · November 18, 2020, 1:38pm

I thought I had this solved - but still seeing errors

PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

wes · April 6, 2021, 3:27pm

I’ve been fighting with the same issue and finally found a solution. Where it presented itself was when I was trying to post an alert to a webhook.

My Setup

Windows Server 2019
Docker for Windows
2 nodes and Kibana - as per the Open Distro Docker install guide

What confused me to no end is that the exact same docker setup running on a different machine worked. I tested on a Windows 10 machine with Docker for Windows and also a mini server running Ubuntu server and Docker.

What finally set me on the right path was connecting to one of the running containers and running
docker exec <container id> curl https://www.google.com

This produced a warning
curl: (60) Peer's Certificate issuer is not recognized. More details here: http://curl.haxx.se/docs/sslcerts.html

This solution on Stack Overflow helped me.

What I did

Open Google Chrome on the Windows Server machine.
Navigate to https://www.google.com
Click on the padlock icon and select “Certificate”
A new windows opens. Click on “Certification Path”
This is the certificate (Fxxxxxxxxx) you need to export in the SO answer step
Export company trusted root certificate with .cer extension. Somthing naming external root certificate
In Windows open “Manage computer certificates”
In my case the certificate was under “Trusted Root Certification Authorities”.
Right click on the certificate and select “All Tasks → Export”. Select where you want to save the *.cer file.
Continue with the steps in the SO answer.

Tips:

To be able to use OpenSSL I installed Ubuntu 20.04 using Windows Subsystem for Linux
I copied the *.PEM file into the container using the command
docker cp <C:\source\path\file.cer> <container id>:/etc/pki/ca-trust/source/anchors/
You’ll need to restart the OpenDistro containers for the changes to take effect.

I realise that copying the *.cer file into the container isn’t the best idea, but was a quick way to see if the solution worked.

Topic		Replies	Views
Disable Webhook TLS / Security? Security	7	921	December 20, 2023
ERR: An unexpected SSLHandshakeException occured: PKIX path building failed: Security	9	1422	April 10, 2023
Java TransportClient insert using the demo certs from the docker images? Security	1	1056	May 7, 2019
PKIX path validation failed Security	3	4442	February 8, 2021
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target Open Source Elasticsearch and Kibana	13	8818	April 16, 2024

TLS/SSL issue - javax.net.ssl.SSLHandshakeException

WARNING: revise all the lines below before you go into production

Related Topics