"SSL Problem PKIX path building failed" & "certificate_unknown" when changing admin certificates

Security
1

Hello wonderful OpenDistro community!

To start, I am running OpenDistro 1.2.0 for ElasticSearch 7.2.0 on CentOS 7 in a single node cluster configuration.

Whilst following the instructions outlined here, to change the admin certificates from the demo certificates, I am receiving the following error when running “securityadmin.sh”:

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:320) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:258) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1324) ~[?:?]
...

The following error in /var/log/elasticsearch/elasticsearch.log occurs concurrently:

javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:285) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:180) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:634) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:295) ~[netty-handler-4.1.35.Final.jar:4.1.35.Final]
...

The parameters that I supplied to the securityadmin.sh script are:

bash /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -rl -h domain.com -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -icl -cert /etc/elasticsearch/domain.com.pem -key /etc/elasticsearch/domain.com-key.pem -cacert /etc/elasticsearch/domain.com.ca-bundle.pem -nhnv

As per the aforementioned guide, I originally tried using certificates signed by a local CA, however, i also purchased an SSL certificates signed by a trusted third party CA and the issue persists.

Given the error in the elasticsearch.log, it’s as though elastic does not trust the certificate/CA despite using a 3rd party trusted root CA.

My elasticsearch.yml configuration is:

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: []
opendistro_security.ssl.transport.pemcert_filepath: esnode.pem
opendistro_security.ssl.transport.pemkey_filepath: esnode-key.pem
opendistro_security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
opendistro_security.ssl.transport.enforce_hostname_verification: false
opendistro_security.ssl.http.enabled: true
opendistro_security.ssl.http.pemcert_filepath: otherdomain.com.pem
opendistro_security.ssl.http.pemkey_filepath: otherdomain.com-key.pem
opendistro_security.ssl.http.pemtrustedcas_filepath: otherdomain.com.root-ca.pem
opendistro_security.allow_unsafe_democertificates: true
opendistro_security.allow_default_init_securityindex: true
opendistro_security.authcz.admin_dn:
   - 'CN=domain.com'
opendistro_security.audit.type: internal_elasticsearch
opendistro_security.enable_snapshot_restore_privilege: true
opendistro_security.check_snapshot_restore_write_privileges: true
opendistro_security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
cluster.routing.allocation.disk.threshold_enabled: false
node.max_local_storage_nodes: 3

I have verified the certificate chain and it is confirmed to be complete without issue and when i verify the subject of the cert, it seems (as far as i can tell) to correspond to the configuration:

 openssl x509 -subject -nameopt RFC2253 -noout -in /etc/elasticsearch/domain.com.pem

returns:

subject= CN=domain.com

When i revert back to using the demo (kirk.pem,kirk-key.pem & root-ca.pem) admin certificates (and amend the “opendistro_security.authcz.admin_dn:” option in the elasticsearch.yml), the securityadmin.sh completes successfully.

I did find this section in the troubleshooting which references the error noticed in the elasticsearch.log, however, there is no IP address added to the certificate and the domain.com.pem has the following SAN:

 X509v3 Subject Alternative Name:
                DNS:domain.com

Do i somehow need to explicitly add the third party or internal Root CA certs to elasticsearch to trust them? If so, any idea how i would go about achieving that?

Any solutions/pointers/advice/suggestions would be greatly welcomed! :beer: :beer:

Thanks so much in advance!
Major

2

I solved my own issue by using the searchguard certificate generation offline tool: Offline TLS Tool | Security for Elasticsearch | Search Guard