Certificate_unknown error and combine two hosts into cluster

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.4.1

Describe the issue:

  1. I have configured my host using docker-compose.yml with two opensearch nodes and one dashboard. I have generated a self signed certificates following this Generating self-signed certificates. I double checked everthing with the certificates and my docker-compose.yml but I keep seeing this error, So I wonder is it because that these are self generated certificates?

  2. Another thing is that i want to combine my hosts nearch02.com.au and nearch03.com.au into one cluster and i am doing it using docker-compose.yml below but when I call the
    /_cat/health API - I see number of hosts one so if you could guide me as to what am I doing wrong here or missing?

Configuration:

services:
   dashboards:
    image: opensearchproject/opensearch-dashboards:2.4.2
    container_name: dashboards.nearch02.com.au
    ports:
      - 5601:5601
    environment:
      - 'OPENSEARCH_HOSTS=["https://nearch02.com.au:9200","https://nearch02.com.au:9201","https://nearch03.com.au:9200","https://nearch03.com.au:9201"]'
    volumes:
      - /etc/opensearch-dashboards/custom_opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml:ro
      - /etc/certs/self-signed/client.pem:/usr/share/opensearch-dashboards/config/client.pem:ro
      - /etc/certs/self-signed/client-key.pem:/usr/share/opensearch-dashboards/config/client-key.pem:ro
      - /etc/opensearch-dashboards/custom_opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml:ro
      - /var/log/search/dashboards/access.log:/usr/share/opensearch-dashboards/access.log

  node-1:
    image: opensearchproject/opensearch:2.4.2
    container_name: node-1.nearch02.com.au
    environment:
      - DISABLE_INSTALL_DEMO_CONFIG=true
      - cluster.name=doc-search
      - node.name=node-1.nearch02.com.au
     - node.attr.host=nearch02.com.au
     - network.bind_host=0.0.0.0
     - network.publish_host=x.x.x.x
     - http.port=9201
     - transport.port=9301
discovery.seed_hosts=nearch02.com.au:9300,nearch02.com.au:9301,nearch03.com.au:9300,nearch03.com.au:9301
      - cluster.initial_master_nodes=node-1.nearch02.com.au,node-2.nearch02.com.au,node-1.nearch03.com.au,node-2.nearch03.com.au
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - /var/lib/search/node-1/data:/usr/share/opensearch/data
      - /var/log/search/node-1:/usr/share/opensearch/logs
      - /etc/certs/self-signed/root-ca.pem:/usr/share/opensearch/config/root-ca.pem:ro
      - /etc/certs/self-signed/node1.pem:/usr/share/opensearch/config/node.pem:ro
      - /etc/certs/self-signed/node1-key.pem:/usr/share/opensearch/config/node-key.pem:ro
      - /etc/certs/self-signed/admin.pem:/usr/share/opensearch/config/admin.pem:ro
      - /etc/certs/self-signed/admin-key.pem:/usr/share/opensearch/config/admin-key.pem:ro
      - /etc/opensearch/custom_opensearch.yml:/usr/share/opensearch/config/opensearch.yml:ro
      - /etc/opensearch/opensearch_auth_config.yml:/usr/share/opensearch/config/opensearch-security/config.yml:ro
      - /etc/opensearch/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml:ro
      - /etc/opensearch/roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml:ro
      - /etc/opensearch/roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml:ro
      - /etc/opensearch/nodes_dn.yml:/usr/share/opensearch/config/opensearch-security/nodes_dn.yml:ro
    ports:
      - 9200:9200
      - 9300:9300
 
Similar configuration for node-2

  node-2:
    image: opensearchproject/opensearch:2.4.2
    container_name: node-2.nearch02.com.au

version: '3'

Relevant Logs or Screenshots:

Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:519) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:458) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.84.Final.jar:4.1.84.Final]


exception caught on transport layer [Netty4TcpChannel{localAddress=/192.19.0.5:9300, remoteAddress=/192.11.134.1:36962}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.84.Final.jar:4.1.84.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:519) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:458) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
2

@Daniyal Could you share the full docker-compose.yml file? Also please share the opensearch.yml file content (custom_opensearch.yml)

3

Hi @pablo, Thank you for your response.

This is my full docker-compose.yml

---
services:
  dashboards.nearch02.com.au:
    container_name: dashboards.nearch02.com.au
    environment:
      OPENSEARCH_HOSTS: '["https://nearch02.com.au:9200","https://nearch02.com.au:9201","https://nearch03.com.au:9200","https://nearch03.com.au:9201"]'
    expose:
    - '5601'
    image: opensearchproject/opensearch-dashboards:2.4.2
    ports:
    - 5601:5601
    volumes:
    - /etc/certs/self-signed/root-ca.pem:/usr/share/opensearch-dashboards/config/root-ca.pem:ro
    - /etc/certs/self-signed/client.pem:/usr/share/opensearch-dashboards/config/client.pem:ro
    - /etc/certs/self-signed/client-key.pem:/usr/share/opensearch-dashboards/config/client-key.pem:ro
    - /etc/opensearch-dashboards/custom_opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml:ro
    - /var/log/search/dashboards/access.log:/usr/share/opensearch-dashboards/access.log

  node-1.nearch02.com.au:
    container_name: node-1.nearch02.com.au
    environment:
    - DISABLE_INSTALL_DEMO_CONFIG=true
    - cluster.name=doc-search
    - node.name=node-1.nearch02.com.au
    - node.attr.host_index_mod=0
    - node.attr.room=hpd
    - node.attr.host=nearch02.com.au
    - network.bind_host=0.0.0.0
    - network.publish_host=x.x.x.x
    - http.port=9200
    - transport.port=9300
    - discovery.seed_hosts=nearch02.com.au:9300,nearch02.com.au:9301,nearch03.com.au:9300,nearch03.com.au:9301
    - cluster.initial_master_nodes=node-1.nearch02.com.au,node-2.nearch02.com.au,node-1.nearch03.com.au,node-2.nearch03.com.au
    - bootstrap.memory_lock=true
   - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    image: opensearchproject/opensearch:2.4.2
    ports:
    - 9200:9200
    - 9300:9300
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        hard: 524287
        soft: 524287
    volumes:
    - /var/lib/search/node-1/data:/usr/share/opensearch/data
    - /var/log/search/node-1:/usr/share/opensearch/logs
    - /etc/certs/self-signed/root-ca.pem:/usr/share/opensearch/config/root-ca.pem:ro
    - /etc/certs/self-signed/node1.pem:/usr/share/opensearch/config/node.pem:ro
    - /etc/certs/self-signed/node1-key.pem:/usr/share/opensearch/config/node-key.pem:ro
    - /etc/certs/self-signed/admin.pem:/usr/share/opensearch/config/admin.pem:ro
    - /etc/certs/self-signed/admin-key.pem:/usr/share/opensearch/config/admin-key.pem:ro
    - /etc/opensearch/custom_opensearch.yml:/usr/share/opensearch/config/opensearch.yml:ro
    - /etc/opensearch/opensearch_auth_config.yml:/usr/share/opensearch/config/opensearch-security/config.yml:ro
    - /etc/opensearch/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml:ro
    - /etc/opensearch/roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml:ro
    - /etc/opensearch/roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml:ro
    - /etc/opensearch/nodes_dn.yml:/usr/share/opensearch/config/opensearch-security/nodes_dn.yml:ro

  node-2.nearch02.com.au:
    container_name: node-2.nearch02.com.au
    environment:
    - DISABLE_INSTALL_DEMO_CONFIG=true
    - cluster.name=doc-search
    - node.name=node-2.nearch02.com.au
    - node.attr.host_index_mod=0
    - node.attr.room=hpd
    - node.attr.host=nearch02.com.au
    - network.bind_host=0.0.0.0
    - network.publish_host=x.x.x.x
    - http.port=9201
    - transport.port=9301
    - discovery.seed_hosts=nearch02.com.au:9300,nearch02.com.au:9301,nearch03.com.au:9300,nearch03.com.au:9301
    - cluster.initial_master_nodes=node-1.nearch02.com.au,node-2.nearch02.com.au,node-1.nearch03.com.au,node-2.nearch03.com.au
    - bootstrap.memory_lock=true
     - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    image: opensearchproject/opensearch:2.4.2
    ports:
    - 9200:9200
    - 9300:9300
    ulimits:
      memlock:
        hard: -1
        soft: -1
      nofile:
        hard: 524287
        soft: 524287
    volumes:
    - /var/lib/search/node-2/data:/usr/share/opensearch/data
    - /var/log/search/node-2:/usr/share/opensearch/logs
    - /etc/certs/self-signed/root-ca.pem:/usr/share/opensearch/config/root-ca.pem:ro
    - /etc/certs/self-signed/node2.pem:/usr/share/opensearch/config/node.pem:ro
    - /etc/certs/self-signed/node2-key.pem:/usr/share/opensearch/config/node-key.pem:ro
    - /etc/certs/self-signed/admin.pem:/usr/share/opensearch/config/admin.pem:ro
    - /etc/certs/self-signed/admin-key.pem:/usr/share/opensearch/config/admin-key.pem:ro
    - /etc/opensearch/custom_opensearch.yml:/usr/share/opensearch/config/opensearch.yml:ro
    - /etc/opensearch/opensearch_auth_config.yml:/usr/share/opensearch/config/opensearch-security/config.yml:ro
    - /etc/opensearch/internal_users.yml:/usr/share/opensearch/config/opensearch-security/internal_users.yml:ro
    - /etc/opensearch/roles.yml:/usr/share/opensearch/config/opensearch-security/roles.yml:ro
    - /etc/opensearch/roles_mapping.yml:/usr/share/opensearch/config/opensearch-security/roles_mapping.yml:ro
    - /etc/opensearch/nodes_dn.yml:/usr/share/opensearch/config/opensearch-security/nodes_dn.yml:ro

version: '3'

And this is the custom_opensearch.yml

---
cluster:
    name: doc-search
    routing:
        allocation:
            awareness:
                attributes: host_index_mod
network:
    host: 0.0.0.0
opendistro_security:
    audit:
        enable_rest: 'true'
        enable_transport: 'true'
plugins:
    security:
        advanced_modules_enabled: 'true'
        allow_default_init_securityindex: 'true'
        allow_unsafe_democertificates: 'false'
        audit:
            type: debug # log4j
        authcz:
            admin_dn:
                [
                    'CN=admin,OU=abc,O=defg,C=hi',
                ]
        cache:
            ttl_minutes: '60'
        disabled: 'false'
        nodes_dn:
            [
                'CN=nearch02.com.au,OU=abc,O=defg,C=hi',
                'CN=nearch03.com.au,OU=abc,O=defg,C=hi',
            ]
        nodes_dn_dynamic_config_enabled: 'true'
        restapi.roles_enabled:
            [
                all_access,
                security_rest_api_access,
            ]
        roles_mapping_resolution: MAPPING_ONLY
        ssl:
            http:
                enabled: 'true'
                clientauth_mode: OPTIONAL
                pemcert_filepath: node.pem
                pemkey_filepath: node-key.pem
                pemtrustedcas_filepath: root-ca.pem
            transport:
                enabled: 'true'
                pemcert_filepath: node.pem
                pemkey_filepath: node-key.pem
                pemtrustedcas_filepath: root-ca.pem
                resolve_hostname: 'false'
                enforce_hostname_verification: 'false'
4

@Daniyal Thanks for sharing the files. Please also share the content of /etc/opensearch-dashboards/custom_opensearch_dashboards.yml

I’ve noticed that shared errors are truncated. Could you share the full error starting from the timestamp?

5

@Daniyal Do you already have a running cluster with nearch03.com.au nodes and want to connect to this cluster? If so, do all nodes nearch03.com.au and nearch02.com.au use the same root-ca.pem?

6

@pablo This is the custom_opensearch_dashbaords.yml

---
logging.dest: /usr/share/opensearch-dashboards/access.log
opensearch:
    pingTimeout: 1500
    requestTimeout: 30000
    shardTimeout: 30000
    ssl:
        certificate: /usr/share/opensearch-dashboards/config/client.pem
        certificateAuthorities: /usr/share/opensearch-dashboards/config/root-ca.pem
        key: /usr/share/opensearch-dashboards/config/client-key.pem
        verificationMode: certificate
        alwaysPresentCertificate: 'true'
opensearchDashboards:
    index: .opensearch_dashboards
opensearch_security:
    auth:
        multiple_auth_enabled: 'true'
        type: ["basicauth", "openid"]
    cookie:
        secure: 'true'
    multitenancy:
        enabled: 'false'
    openid:
        base_redirect_url: https://nearch02.com.au:5601
        client_id: id
        client_secret: secret
        connect_url: https://someurl
        scope: openid profile email preferred_username
server:
    host: 0.0.0.0
    port: 5601
    ssl:
        enabled: 'true'
        clientAuthentication: none
        certificate: /usr/share/opensearch-dashboards/config/client.pem
        key: /usr/share/opensearch-dashboards/config/client-key.pem
        certificateAuthorities: /usr/share/opensearch-dashboards/config/root-ca.pem

and this is the requested error from timestamp.



[2024-11-01T14:49:58,556][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1.nearch02.com.au] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:519) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:458) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.84.Final.jar:4.1.84.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-11-01T14:49:58,557][WARN ][o.o.t.TcpTransport       ] [node-1.nearch02.com.au] exception caught on transport layer [Netty4TcpChannel{localAddress=/x.x.x.x:9300, remoteAddress=/x.x.x.x:34938}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:280) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.84.Final.jar:4.1.84.Final]
        at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:358) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:296) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1343) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[netty-handler-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:519) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:458) ~[netty-codec-4.1.84.Final.jar:4.1.84.Final]
7

@pablo So I have two hosts nearch02.com.au and nearch03.com.au, first I configured them individually each with two opensearch nodes and one dashboard using docker-compose.yml.
I could reach the dashboard but there was certificate_unknown error. I couldn’t figured it out why is that so I continued on to combine them into one cluster and the current configuration shared here as what I am trying. But when i ran the docker-compose.yml on both hosts and then run API to check no of nodes it is showing just one. I don’t know if i have to do something on the networking part or is it due to the certificate_unknown error. I generated a separate certificates but the root CN is the same.

8

@Daniyal According to the errors, the OpenSearch nodes can’t communicate with each other on the Transport layer (9300-9400) due to an unknown certificate.

The OpenSearch node will report this error when the other OpenSearch node in the same cluster is using a certificate signed by a different Root CA certificate (ssl.transport.pemtrustedcas_fielpath).

You can concatenate these RootCA in all of the nodes and keep using existing node certificates.

Please also check the node and root CA certificates in each running OpenSearch node. Maybe the file mapping didn’t properly work and the node is using demo certificates.