Certificates not validated with CA certificates in truststore for CCA

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Opensearch 2.11

Describe the issue:

I have 2 different clusters and I’m trying to configure cross cluster replication.
Now, each of the clusters have certificates signed by a different CA. I imported the CA certificates of each other in a truststore and included the trustore in the deployments, besides including the CA certificate for each cluster in the pemtrustedcas_filepath.

In the leader cluster I see the error certificate_unknown, I interpret it fails to validate the certificate based on the CA certificates in the truststore, even when the CA is included in the truststore.
In the follower cluster I get the error unable to find valid certification path to requested target

I was able to setup CCR when the certificates for the different clusters are signed by the same CA and the CA certificate is provided in pemtrustedcas_filepath, but when the clusters have certificates signed by different CAs, I cannot get it to work.

Is there any limitation when truststore is not used to validate remote certificates?
I’m using truststore to be able to validate the certificate for the LDAP server, and that works ok.

I cannot either use securityadmin.sh script with admin certificates signed by a different CA than the one CA certificate configured in pemtrustedcas_filepath, even when the CA certificate is in the truststore.

Configuration:

        ssl:
          transport:
            # If security is enabled, transport security must be enabled
            enabled: true
            pemcert_filepath: node-certs/node.crt
            pemkey_filepath: node-certs/node.key
            pemtrustedcas_filepath: root-ca-cert/ca.crt
            truststore_filepath: /usr/share/opensearch/config/ca-certs/cacerts.jks
            truststore_password: ${TRUSTSTORE_PASSWORD}
            enforce_hostname_verification: false
          http:
            enabled: false
            pemcert_filepath: node-certs/node.crt
            pemkey_filepath: node-certs/node.key
            pemtrustedcas_filepath: root-ca-cert/ca.crt
            truststore_filepath: /usr/share/opensearch/config/ca-certs/cacerts.jks
            truststore_password: ${TRUSTSTORE_PASSWORD}
            enforce_hostname_verification: false
            clientauth_mode: OPTIONAL
        ssl_cert_reload_enabled: true
        allow_unsafe_democertificates: false
        allow_default_init_securityindex: true

Relevant Logs or Screenshots:

Leader Node

        [2025-02-03T21:40:45,926][WARN ][o.o.t.TcpTransport       ] [opensearch-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.42.14.72:9300, remoteAddress=/10.42.2.61:45458}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.100.Final.jar:4.1.100.Final]

Follower node

 [2025-02-03T21:40:45,716][WARN ][o.o.t.TcpTransport       ] [opensearch-dev3-master-0] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.42.1.207:37488, remoteAddress=opensearch-dev-ccr.blue.k8s.lab.com/10.56.160.116:9300}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.100.Final.jar:4.1.100.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.100.Final.jar:4.1.100.Final]

Hi @Julio, great question, this issue usually occurs when the ca files listed in pemtrustedcas_filepath do not contain both CA’s.

I would recommend to concatenate both CA files into one and provide this new file in pemtrustedcas_filepath. (On both side of the replication)

Also I assume you updated the plugins.security.nodes_dn appropriately when testing this. You can find more details here

As a last note, using truststore and pem files simultaneously is rather unconventional and it might have potential to interfere with the workflow, perhaps only one can be used in your use case?