Replace DEMO SSL certificate for Production CA DigiCert wildcard

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Verision: docker compose using opensearchproject/opensearch:latest
At the time I am writing the version is 2.6.0

Describe the issue:
I need to install the certificate from DigiCert, which provides me three files only.
We have a wildcard certificate, and the files are:

  • DigiCertCA2.pem (Intermediate CA)
  • star_company_com.pem
  • TrustedRoot.pem (Root CA)

How Can I replace the demo certificates to use the CA one?
Or even better how can I sign the admin, client, node certificate with my root CA if I don’t have the private key ?

I tried to follow the self-signed guide to see if it worked but I still get an SSL exception:

Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

Configuration:
docker-compose.yml

services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - ./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./certs/admin.pem:/usr/share/opensearch/config/admin.pem
      - ./certs/admin-key.pem:/usr/share/opensearch/config/admin-key.pem
      - ./certs/node.pem:/usr/share/opensearch/config/node.pem
      - ./certs/node-key.pem:/usr/share/opensearch/config/node-key.pem
      - ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./certs/admin.pem:/usr/share/opensearch/config/admin.pem
      - ./certs/admin-key.pem:/usr/share/opensearch/config/admin-key.pem
      - ./certs/node.pem:/usr/share/opensearch/config/node.pem
      - ./certs/node-key.pem:/usr/share/opensearch/config/node-key.pem
      - ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

custom-opensearch.yml

network.host: 0.0.0.0
plugins.security.ssl.transport.pemcert_filepath: node.pem
plugins.security.ssl.transport.pemkey_filepath: node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node.pem
plugins.security.ssl.http.pemkey_filepath: node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=ADMIN,OU=IT,O=COMPANY,L=CHICAGO,ST=ILLINOIS,C=US'
plugins.security.nodes_dn:
  - 'CN=COMPANY.COM,OU=IT,O=COMPANY,L=CHICAGO,ST=ILLINOIS,C=US'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]

@fdinardo What’s the star_company_com.pem? Is that a node certificate?
If yes, you should also receive a key with that cert.

In regards to signing the admin certs, this is done by DigiCert as they are Trusted CA and can sign your CSRs.
If you’d like to use self-signed certificates for the admin certificates you can do that by adding self-signed RootCAs of your admin cert to the file defined in plugins.security.ssl.http.pemtrustedcas_filepath

Hi @pablo ,
Thanks for the reply.
I am not familiar with all these certificates theory…
When I download the certificates from the DigiCert portal, I don’t have any KEY back.
According to Digicert, they are not going to provide any private Key (which makes sense), but I am not sure where I can grab, or generate it.

To give more context, my issue is that I am able to connect to the server running docker, and get access to the dashboard on port 5601, the same for port 9200, but since the https://:9200 has no Certificates, the browser gives me a warning, and when I try to connect to it through C# it fails since it is unsecure…

This is what I get from the Digicert portal:

where:

  • Certificate is the star_company_com.pem certificate.
  • Intermediate Certificate is TrustedRoot.pem
  • Root Certficate is DigiCertCA2.pem

I followed the self-signed Certificate Guide bypassing completely the Certificates from DigiCert, but I still get the same warning in the browser.

I am sorry, I am not really familiar with all these set up.

@pablo
Good news, I talked to our System Administrator and they provided me the private key!

So, if I got this correctly, I can use the DigiCertCA2.pem file and the key to sign the other certificates ?

then I have to include the DigiCertCA2.pem here: plugins.security.ssl.transport.pemtrustedcas_filepath.

OK, I think we made some progress here.
I was able to use the right certificate over https for port 9200.
The problem is that I get this error on the browser:
OpenSearch Security not initialized.
In the logs I get this error:

opensearch-node2       | [2023-04-24T17:52:24,656][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [opensearch-node2] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
opensearch-node2       | javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
opensearch-node2       |        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
opensearch-node2       |        at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]
opensearch-node2       |        at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]
opensearch-node2       |        at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
opensearch-node2       |        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
opensearch-node2       |        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-node2       | Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
opensearch-node2       |        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
opensearch-node2       |        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
opensearch-node2       |        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
opensearch-node2       |        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
opensearch-node2       |        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
opensearch-node2       |        ... 30 more
opensearch-node2       | Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
opensearch-node2       |        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
opensearch-node2       |        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
opensearch-node2       |        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
opensearch-node2       |        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
opensearch-node2       |        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
opensearch-node2       |        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
opensearch-node2       |        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
opensearch-node2       |        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
opensearch-node2       |        ... 30 more
opensearch-node2       | [2023-04-24T17:52:24,660][WARN ][o.o.t.TcpTransport       ] [opensearch-node2] exception caught on transport layer [Netty4TcpChannel{localAddress=/172.20.0.3:59022, remoteAddress=opensearch-node1/172.20.0.4:9300}], closing connection
opensearch-node2       | io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-node2       | Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
opensearch-node2       |        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
opensearch-node2       |        at sun.security.ssl.TransportContext.fatal(TransportContext.java:371) ~[?:?]
opensearch-node2       |        at sun.security.ssl.TransportContext.fatal(TransportContext.java:314) ~[?:?]
opensearch-node2       |        at sun.security.ssl.TransportContext.fatal(TransportContext.java:309) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
opensearch-node2       |        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
opensearch-node2       |        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        ... 16 more
opensearch-node2       | Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
opensearch-node2       |        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
opensearch-node2       |        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
opensearch-node2       |        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
opensearch-node2       |        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
opensearch-node2       |        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
opensearch-node2       |        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
opensearch-node2       |        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        ... 16 more
opensearch-node2       | Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
opensearch-node2       |        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
opensearch-node2       |        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
opensearch-node2       |        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
opensearch-node2       |        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
opensearch-node2       |        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
opensearch-node2       |        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
opensearch-node2       |        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
opensearch-node2       |        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
opensearch-node2       |        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
opensearch-node2       |        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1277) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1264) ~[?:?]
opensearch-node2       |        at java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
opensearch-node2       |        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1209) ~[?:?]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1549) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1395) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node2       |        ... 16 more
opensearch-node1       | [2023-04-24T17:52:24,665][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [opensearch-node1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
opensearch-node1       | javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
opensearch-node1       |        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
opensearch-node1       |        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
opensearch-node1       |        at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
opensearch-node1       |        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
opensearch-node1       |        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
opensearch-node1       |        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
opensearch-node1       |        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-node1       | [2023-04-24T17:52:24,666][WARN ][o.o.t.TcpTransport       ] [opensearch-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=/172.20.0.4:9300, remoteAddress=/172.20.0.3:59022}], closing connection
opensearch-node1       | io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
opensearch-node1       |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-node1       | Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
opensearch-node1       |        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
opensearch-node1       |        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
opensearch-node1       |        at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
opensearch-node1       |        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
opensearch-node1       |        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
opensearch-node1       |        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
opensearch-node1       |        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
opensearch-node1       |        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]
opensearch-node1       |        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.87.Final.jar:4.1.87.Final]

So I am using the TrustedRoot.pem as root-ca.pem and for node-key.pem and node.pem I am using the private key and star_company_com.pem accordingly.

I think the problem here is with the admin certificates because when I land on the 9200 port through the browser I see the certificate from DigiCert.

@fdinardo That should be correct. You should see the node’s certificate signed by the Intermediate certificate.

Did you verify the files inside the containers? Do you get the correct openserach.yml and cert/key files in both OpenSearch containers?

Have you noticed any read file permission errors in the OpenSearch logs during the startup?

Did you add the RootCA of the admin certificate to the root-ca.pem?

If the RootCAs are different then the root-ca.pem should look like the below example.

-----BEGIN CERTIFICATE-----
<node RootCA certificate>
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<admin RootCA certificate>
-----END CERTIFICATE-----
1 Like

Yes, I checked both nodes, and the files are there:

$ docker exec ec3975eb906b ls -l /usr/share/opensearch/config
total 64
-rw------- 1 opensearch opensearch 1708 Apr 25 09:28 admin-key.pem
-rw-r--r-- 1 opensearch opensearch 1204 Apr 25 09:28 admin.pem
-rw-rw---- 1 opensearch opensearch 2660 Mar 20 17:32 jvm.options
drwxr-x--- 2 opensearch opensearch 4096 Feb 23 19:20 jvm.options.d
-rw-rw---- 1 opensearch opensearch  285 Mar 20 17:32 log4j2.properties
-rw-r--r-- 1 opensearch opensearch 1675 Apr 25 09:23 node-key.pem
-rw-r--r-- 1 opensearch opensearch 2446 Apr 25 09:23 node.pem
drwxr-x--- 2 opensearch opensearch 4096 Feb 24 19:22 opensearch-notifications
drwxr-x--- 2 opensearch opensearch 4096 Feb 24 19:22 opensearch-notifications-core
drwxr-x--- 2 opensearch opensearch 4096 Feb 24 19:22 opensearch-observability
drwxr-x--- 2 opensearch opensearch 4096 Feb 24 19:22 opensearch-performance-analyzer
drwxr-x--- 2 opensearch opensearch 4096 Feb 24 19:22 opensearch-reports-scheduler
drwxr-x--- 2 opensearch opensearch 4096 Feb 24 19:22 opensearch-security
-rw-rw---- 1 opensearch opensearch  196 Apr 25 09:44 opensearch.keystore
-rw-r--r-- 1 opensearch opensearch 1557 Apr 25 09:28 opensearch.yml
-rw-r--r-- 1 opensearch opensearch 2664 Apr 25 09:28 root-ca.pem

Yes, I have some warnings:

opensearch-node2       | [2023-04-25T09:29:35,532][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
opensearch-node2       | [2023-04-25T09:29:35,541][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
opensearch-node2       | [2023-04-25T09:29:35,541][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
opensearch-node2       | [2023-04-25T09:29:35,543][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/node-key.pem has insecure file permissions (should be 0600)
opensearch-node2       | [2023-04-25T09:29:35,543][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/admin.pem has insecure file permissions (should be 0600)
opensearch-node2       | [2023-04-25T09:29:35,544][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node2] File /usr/share/opensearch/config/node.pem has insecure file permissions (should be 0600)

opensearch-node1       | [2023-04-25T09:29:36,208][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] Directory /usr/share/opensearch/config has insecure file permissions (should be 0700)
opensearch-node1       | [2023-04-25T09:29:36,212][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] File /usr/share/opensearch/config/opensearch.yml has insecure file permissions (should be 0600)
opensearch-node1       | [2023-04-25T09:29:36,213][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] File /usr/share/opensearch/config/root-ca.pem has insecure file permissions (should be 0600)
opensearch-node1       | [2023-04-25T09:29:36,214][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] File /usr/share/opensearch/config/node-key.pem has insecure file permissions (should be 0600)
opensearch-node1       | [2023-04-25T09:29:36,216][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] File /usr/share/opensearch/config/admin.pem has insecure file permissions (should be 0600)
opensearch-node1       | [2023-04-25T09:29:36,217][WARN ][o.o.s.OpenSearchSecurityPlugin] [opensearch-node1] File /usr/share/opensearch/config/node.pem has insecure file permissions (should be 0600)

Yes, I combined trusted-root.pem and the root-ca-admin.pem into the root-ca.pem file:

cat trusted-root.pem root-ca-admin.pem > root-ca.pem

@pablo I really want to thank you, coz you drove me to the right direction.

Finally I was able to get it to work with only one Exception:

opensearch-node2       | [2023-04-25T12:02:53,665][INFO ][o.o.c.c.JoinHelper       ] [opensearch-node2] failed to join {opensearch-node1}{rpJrJk1XQ5itqukmivi_Vg}{UgcfUXIyR4CQV2kJtoMk5w}{192.168.16.4}{192.168.16.4:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={opensearch-node2}{oLDXZoDSQR-TQ0puoKZcsQ}{V1yPQBpeSFK499yxp_RmIg}{192.168.16.2}{192.168.16.2:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={opensearch-node2}{oLDXZoDSQR-TQ0puoKZcsQ}{V1yPQBpeSFK499yxp_RmIg}{192.168.16.2}{192.168.16.2:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={opensearch-node1}{rpJrJk1XQ5itqukmivi_Vg}{UgcfUXIyR4CQV2kJtoMk5w}{192.168.16.4}{192.168.16.4:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
opensearch-node2       | org.opensearch.transport.RemoteTransportException: [opensearch-node1][192.168.16.4:9300][internal:cluster/coordination/join]
opensearch-node2       | Caused by: org.opensearch.cluster.coordination.CoordinationStateRejectedException: incoming term 1 does not match current term 2
opensearch-node2       |        at org.opensearch.cluster.coordination.CoordinationState.handleJoin(CoordinationState.java:256) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:1179) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at java.util.Optional.ifPresent(Optional.java:178) ~[?:?]
opensearch-node2       |        at org.opensearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:647) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:610) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.action.ActionListener$1.onResponse(ActionListener.java:80) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.transport.ClusterConnectionManager.connectToNode(ClusterConnectionManager.java:138) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.transport.TransportService.connectToNode(TransportService.java:450) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.transport.TransportService.connectToNode(TransportService.java:430) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:592) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.cluster.coordination.JoinHelper.lambda$new$1(JoinHelper.java:190) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceivedDecorate(SecuritySSLRequestHandler.java:192) ~[?:?]
opensearch-node2       |        at org.opensearch.security.transport.SecurityRequestHandler.messageReceivedDecorate(SecurityRequestHandler.java:278) ~[?:?]
opensearch-node2       |        at org.opensearch.security.ssl.transport.SecuritySSLRequestHandler.messageReceived(SecuritySSLRequestHandler.java:152) ~[?:?]
opensearch-node2       |        at org.opensearch.security.OpenSearchSecurityPlugin$7$1.messageReceived(OpenSearchSecurityPlugin.java:659) ~[?:?]
opensearch-node2       |        at org.opensearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:108) ~[?:?]
opensearch-node2       |        at org.opensearch.performanceanalyzer.transport.PerformanceAnalyzerTransportRequestHandler.messageReceived(PerformanceAnalyzerTransportRequestHandler.java:43) ~[?:?]
opensearch-node2       |        at org.opensearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:106) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:453) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.6.0.jar:2.6.0]
opensearch-node2       |        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
opensearch-node2       |        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
opensearch-node2       |        at java.lang.Thread.run(Thread.java:833) [?:?]
opensearch-node2       | [2023-04-25T12:02:53,747][INFO ][o.o.c.s.MasterService    ] [opensearch-node2] elected-as-cluster-manager ([2] nodes joined)[{opensearch-node2}{oLDXZoDSQR-TQ0puoKZcsQ}{V1yPQBpeSFK499yxp_RmIg}{192.168.16.2}{192.168.16.2:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, {opensearch-node1}{rpJrJk1XQ5itqukmivi_Vg}{UgcfUXIyR4CQV2kJtoMk5w}{192.168.16.4}{192.168.16.4:9300}{dimr}{shard_indexing_pressure_enabled=true} elect leader, _BECOME_CLUSTER_MANAGER_TASK_, _FINISH_ELECTION_], term: 2, version: 1, delta: cluster-manager node changed {previous [], current [{opensearch-node2}{oLDXZoDSQR-TQ0puoKZcsQ}{V1yPQBpeSFK499yxp_RmIg}{192.168.16.2}{192.168.16.2:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, added {{opensearch-node1}{rpJrJk1XQ5itqukmivi_Vg}{UgcfUXIyR4CQV2kJtoMk5w}{192.168.16.4}{192.168.16.4:9300}{dimr}{shard_indexing_pressure_enabled=true}}
opensearch-node1       | [2023-04-25T12:02:53,816][INFO ][o.o.c.c.JoinHelper       ] [opensearch-node1] failed to join {opensearch-node1}{rpJrJk1XQ5itqukmivi_Vg}{UgcfUXIyR4CQV2kJtoMk5w}{192.168.16.4}{192.168.16.4:9300}{dimr}{shard_indexing_pressure_enabled=true} with JoinRequest{sourceNode={opensearch-node1}{rpJrJk1XQ5itqukmivi_Vg}{UgcfUXIyR4CQV2kJtoMk5w}{192.168.16.4}{192.168.16.4:9300}{dimr}{shard_indexing_pressure_enabled=true}, minimumTerm=0, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={opensearch-node1}{rpJrJk1XQ5itqukmivi_Vg}{UgcfUXIyR4CQV2kJtoMk5w}{192.168.16.4}{192.168.16.4:9300}{dimr}{shard_indexing_pressure_enabled=true}, targetNode={opensearch-node1}{rpJrJk1XQ5itqukmivi_Vg}{UgcfUXIyR4CQV2kJtoMk5w}{192.168.16.4}{192.168.16.4:9300}{dimr}{shard_indexing_pressure_enabled=true}}]}
opensearch-node1       | org.opensearch.transport.RemoteTransportException: [opensearch-node1][192.168.16.4:9300][internal:cluster/coordination/join]
opensearch-node1       | Caused by: org.opensearch.cluster.coordination.CoordinationStateRejectedException: became follower
opensearch-node1       |        at org.opensearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.lambda$close$3(JoinHelper.java:570) [opensearch-2.6.0.jar:2.6.0]
opensearch-node1       |        at java.util.HashMap$Values.forEach(HashMap.java:1065) [?:?]
opensearch-node1       |        at org.opensearch.cluster.coordination.JoinHelper$CandidateJoinAccumulator.close(JoinHelper.java:570) [opensearch-2.6.0.jar:2.6.0]
opensearch-node1       |        at org.opensearch.cluster.coordination.Coordinator.becomeFollower(Coordinator.java:745) [opensearch-2.6.0.jar:2.6.0]
opensearch-node1       |        at org.opensearch.cluster.coordination.Coordinator.onFollowerCheckRequest(Coordinator.java:344) [opensearch-2.6.0.jar:2.6.0]
opensearch-node1       |        at org.opensearch.cluster.coordination.FollowersChecker$2.doRun(FollowersChecker.java:228) [opensearch-2.6.0.jar:2.6.0]
opensearch-node1       |        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-2.6.0.jar:2.6.0]
opensearch-node1       |        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-2.6.0.jar:2.6.0]
opensearch-node1       |        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
opensearch-node1       |        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
opensearch-node1       |        at java.lang.Thread.run(Thread.java:833) [?:?]

I think this exception is related to the node2 not able to join the cluster, but I am not sure why.

For other people looking for this solution, I am going to add here the steps that worked for me:

  • I created self-signed certificates for the admin, hence a root-ca-admin.pem following this guide on the documentation
  • I added to the root-ca.pem the certificates from Digicert, plus the root-ca-admin.pem:
$ cat DigiCertCA2.pem TrustedRoot.pem root-ca-admin.pem > root-ca.pem
  • I used star_company_com.pem as the node.pem certificates and wildcard_company_com_key.pem as the node-key.pem.

  • Next step was to give the right permissions and set the right owners to the files we want to add to the volumes section of docker-compose.yml:

$ chown 1000:1000 custom-opensearch.yml node-key.pem node.pem admin-key.pem admin.pem root-ca.pem
$ chmod 0600 custom-opensearch.yml node-key.pem node.pem admin-key.pem admin.pem root-ca.pem
  • I also updated my custom-opensearch.yml to add the another entry (for the second node) to this directive:
plugins.security.nodes_dn:
  - 'CN=*.company.com,O=Company\, Inc.,L=CITY,ST=STATE,C=US'
  - 'CN=*.company.com,O=Company\, Inc.,L=CITY,ST=STATE,C=US'

Make sure to properly escape special characters like commas( \, ).

It is possible that, the second line, did create the issue with the second node not being able to join the cluster, I am going to test again by removing the second entry.

Hope this could help other people in my situation.
Thanks

Question for you guys. I have a multidomain cert where SAN includes all the names of my nodes, and CN has something generic. The cert is issued by our Private CA. So then I have Intermediate and Root CA’s.

How do I make the following work, I am getting Transport Client Authentication no longer supported

plugins.security.ssl.transport.pemcert_filepath: (multi-domain cert)
plugins.security.ssl.transport.pemkey_filepath: (key matching the cert)
plugins.security.ssl.transport.pemtrustedcas_filepath: (Intermediate and Root certs in chain)

If so I am still getting the error

I also have this

plugins.security.nodes_dn:

  - 'CN=opensearch.example.pvt,OU=Enterprise SSL,O=Example\, Inc'
  - 'CN=opensearch.exmaple.pvt,OU=Enterprise SSL,O=Example\, Inc'

Hi @stecino ,

  • How many nodes do you have ?
  • Is the admin cert self-signed ?

If you still get the error related to the second node not being able to join the cluster, pay attention on the type of that error, if it is [INFO] or [DEBUG] or [WARNING], it is normal because the exception is handled.
Check this discussion for more information.

If the error is the one in the OP, than it means that either your certs are not correct or the plugins.security.nodes_dn: is not correct.

Also, make sure you included this plugins.security.authcz.admin_dn: that matches the certificate you created, if it was self-signed.

@fdinardo I am starting with two nodes. Admin cert is issued by the same Issuer as the cert used for the nodes. So I have same Intermediate and Root certs for node certs and admin cert. Where do I add my admin cert and key, besides making a reference to it in the plugins.security.authcz.admin_dn