SSL Certificate Error "unknown_certificate"

michael.anderton · May 31, 2022, 5:48pm

Can anyone help me diagnose this error?

“Received fatal alert: certificate_unknown”

I am not sure what certificate it is referring to and there is no other information with it that would specify. I have verified that my root cert and client cert/key are valid and contain the entire chain. I am using certificates created with the CA on our Domain Controller. I have converted the .cer files to .pem and the key to pkcs8. I am not sure what else to do.

pablo · May 31, 2022, 7:33pm

@michael.anderton Where do you see these errors?
Could you share your config.yml file?

michael.anderton · May 31, 2022, 7:53pm

@pablo

I am using docker-compose. I see the errors in int node1 logs as I run docker-compose up. I dont have a config.yml file but I overwrite the opensearch.yml file in /usr/share/opensearch/config/opensearch.yml with a yml file that is in a config directory. My config directory is in the same directory as my docker-compose file. Below is my opensearch.yml file. Please let me know if more info is necessary.

---
cluster.name: docker-cluster

compatibility.override_main_response_version: true

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# # minimum_master_nodes need to be explicitly set when bound on a public IP
# # set to 1 to allow single node clusters
# discovery.zen.minimum_master_nodes: 1

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# discovery.type: single-node


######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: opensearch-cert.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root.pem
plugins.security.ssl.transport.pemkey_filepath: opensearch-key.pk8
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: opensearch-cert.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root.pem
plugins.security.ssl.http.pemkey_filepath: opensearch-key.pk8
#plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
#plugins.security.authcz.admin_dn:
#  - CN=kirk,OU=client,O=client,L=test, C=de

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".op$node.max_local_storage_nodes: 3"]
######## End OpenSearch Security Demo Configuration ########

Gsmitt · June 1, 2022, 3:03am

Hello

Insure the correct certificates are in the your keystore. I also found this documentation below, It might help.

pablo · August 8, 2022, 12:40pm

@michael.anderton Have you found a solution to your issue?
If not, could you share your docker-compose.yml?

michael.anderton · August 10, 2022, 2:04pm

@pablo
I seem to have ironed out the unknown cert issue but now dashboards is giving me the following
‘’’
{“type”:“log”,“@timestamp”:“2022-08-10T13:55:01Z”,“tags”:[“error”,“opensearch”,“data”],“pid”:1,“message”:“[ConnectionError]: write EPROTO 140016787933056:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:…/deps/openssl/openssl/ssl/record/ssl3_record.c:332:\n”}
‘’’

Does each node need its own certificate? I have 2 nodes, dashboards, and logstash running. Does that mean I need 4 separate certificates and keys?

pablo · August 10, 2022, 3:52pm

@michael.anderton Could you share the following files?

config.yml
opensearch_dashboards.yml
docker-compose.yml

You should be fine running OpenSearch cluster on the single cert. However, I would recommend using separate certs.

michael.anderton · August 10, 2022, 4:03pm

@pablo
EDIT: I just created and attached an opensearch-dashboards.yml file (below). I am still unsure where I could find config.yml

I am running opensearch with docker-compose, I do not have a config.yml or opensearch_dashboards.yml. I am not sure if they exist inside of the image.

docker-compose.yml

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - ./config/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./cert.pem:/usr/share/opensearch/config/cert.pem
      - ./key.pem:/usr/share/opensearch/config/key.pem
      - ./root.pem:/usr/share/opensearch/config/root.pem
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./config/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./cert.pem:/usr/share/opensearch/config/cert.pem
      - ./key.pem:/usr/share/opensearch/config/key.pem
      - ./root.pem:/usr/share/opensearch/config/root.pem
    networks:
      - opensearch-net

  logstash:
    image: opensearchproject/logstash-oss-with-opensearch-output-plugin:latest
    container_name: logstash
    ports:
      - 0.0.0.0:5044:5044/udp
      - 0.0.0.0:514:514/udp
    volumes:
      - ./logstash.conf/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    networks:
      - opensearch-net


  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

opensearch-dashboards.yml

opensearch.hosts: [https://localhost:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersWhitelist: [authorization, securitytenant]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false
server.host: '0.0.0.0'

pauchamb · August 10, 2022, 4:46pm

Hey @michael.anderton

One thing to try is make sure in your opensearch.yml file, you have the following settings configured:

plugins.security.authcz.admin_dn:
  - [ADMIN CERT SUBJECT]
plugins.security.nodes_dn:
 - [CERT SUBJECT OF NODE X]
 - [CERT SUBJECT OF NODE Y]
 - [CERT SUBJECT OF NODE Z]

Also, make sure that the subjects follow the proper formatting → you can use the below code get the subject of a given cert in the right format.

openssl x509 -subject -nameopt RFC2253 -noout -in ../path/to/file.pem

Hopefully this helps!

michael.anderton · August 10, 2022, 6:25pm

@pauchamb
I do not have a certificate for an admin as of now, but I did add the nodes_dn. Does each node have to have a separate certificate if I am running opensearch in docker?

I am watching the logs for all of my nodes with certificates enabled:
node1 says cert not permitted for client auth
node2 says certificate unknown
logstash says it cant find a valid certification path to node2

pablo · August 11, 2022, 9:19am

@michael.anderton Your docker-compose.yml certificates mappings are as follows

    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./config/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./cert.pem:/usr/share/opensearch/config/cert.pem
      - ./key.pem:/usr/share/opensearch/config/key.pem
      - ./root.pem:/usr/share/opensearch/config/root.pem

However, your opensearch.yml file expects the following.

plugins.security.ssl.transport.pemcert_filepath: opensearch-cert.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root.pem
plugins.security.ssl.transport.pemkey_filepath: opensearch-key.pk8
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: opensearch-cert.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root.pem
plugins.security.ssl.http.pemkey_filepath: opensearch-key.pk8

In a result, your OpenSearch node fails to use your custom certificates.

All security plugin config files are either in

/usr/share/opensearch/plugins/opensearch-security/securityconfig

up to version 1.3

or

/usr/share/opensearch/config/opensearch-security

from version 2.x

Since you’ve not defined config.yml in mappings, OpenSearch will use default config.yml with basic auth enabled.

michael.anderton · August 11, 2022, 5:05pm

@pablo

Are you saying that my certificate files need to be named as you stated? “opensearch-cert.pem” and that cert.pem will not work?

pablo · August 12, 2022, 10:11am

@michael.anderton In your opensearch.yml you’ve used the following file naming.

opensearch-cert.pem
opensearch-key.pk8
root.pem

In docker-compose.yml you’ve mapped

cert.pem to cert.pem in the OpenSearch container
key.pem to key.pem in the OpenSearch container
root.pem to root.pem in the OpenSearch container

As a result, the OpenSearch container can’t access opensearch-cert.pem and opensearch-key.pk8.

Your mapping should be as the below.

    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./config/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./cert.pem:/usr/share/opensearch/config/opensearch-cert.pem
      - ./key.pem:/usr/share/opensearch/config/opensearch-key.pk8
      - ./root.pem:/usr/share/opensearch/config/root.pem

Also, check the opensearch.yml in the running OpenSearch container and confirm it matches your custom opensearch.yml file.

docker exec <container_name_or_ID> cat config/opensearch.yml

michael.anderton · August 15, 2022, 3:19pm

@pablo

The cert unknown error seems to have disappeared when I disabled hostname verification. I am now having an issue with client authentication. When I create a cert with my windows CA with only server authentication I get an error that says they key does not support client authentication, when I create on with client authentication I get an error on node 1 that says client authentication is no longer supported. Both nodes saying that the TLS handshake is failing. It is my understanding that if hostname verification is disabled (false) then the entire cluster can use one certificate for all purposes. The certs I have been trying are using the hostname of the ubuntu host with the host IP and opensearch is running in docker. I have 2 nodes, dashboards, and logstash.

pablo · August 15, 2022, 4:36pm

@michael.anderton Have you fixed that volume mapping?

pauchamb · August 15, 2022, 4:52pm

The “client authentication is no longer supported” error goes back to the plugins.security.nodes_dn: setting that I had to use in my config. Without the correct certificates listed, the clients would fail to talk to each other within the cluster.

michael.anderton · August 15, 2022, 6:12pm

@pablo @pauchamb @nateynate

I believe the plugins.security.nodes_dn: fixed the client auth problem. I am putting my docker-compose.yml and opensearch.yml contents for review because now I can get the stack to run and I can log in with https but node 1 is giving me the below error and warning. Node 2 is not giving any error.

[ERROR][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport] [opensearch-node1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

[WARN ][o.o.h.AbstractHttpServerTransport] [opensearch-node1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/192.168.128.4:9200, remoteAddress=/192.168.10.250:61427}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

Docker-compose.yml

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - ./config/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./cert.pem:/usr/share/opensearch/config/cert.pem
      - ./key.pem:/usr/share/opensearch/config/key.pem
      - ./root.pem:/usr/share/opensearch/config/root.pem
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net

  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./config/opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./cert.pem:/usr/share/opensearch/config/cert.pem
      - ./key.pem:/usr/share/opensearch/config/key.pem
      - ./root.pem:/usr/share/opensearch/config/root.pem
    networks:
      - opensearch-net

  logstash:
    image: opensearchproject/logstash-oss-with-opensearch-output-plugin:latest
    container_name: logstash
    ports:
      - 0.0.0.0:5044:5044/udp
      - 0.0.0.0:514:514/udp
    volumes:
      - ./logstash.conf/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
      - ./config/opensearch.yml:/usr/share/logstash/config/opensearch.yml
      - ./cert.pem:/usr/share/logstash/config/cert.pem
      - ./key.pem:/usr/share/logstash/config/key.pem
      - ./root.pem:/usr/share/logstash/config/root.pem
    networks:
      - opensearch-net


  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    volumes:
      - ./config/opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
      - ./cert.pem:/usr/share/opensearch-dashboards/config/cert.pem
      - ./key.pem:/usr/share/opensearch-dashboards/config/key.pem
      - ./root.pem:/usr/share/opensearch-dashboards/config/root.pem
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]'
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

Opensearch.yml (Certificate SAN info removed)

---
cluster.name: opensearch-cluster

compatibility.override_main_response_version: true

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# # minimum_master_nodes need to be explicitly set when bound on a public IP
# # set to 1 to allow single node clusters
# discovery.zen.minimum_master_nodes: 1

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# discovery.type: single-node


######## Start OpenSearch Security Demo Configuration ########
# WARNING: revise all the lines below before you go into production

plugins.security.ssl.transport.pemcert_filepath: cert.pem
plugins.security.ssl.transport.pemkey_filepath: key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: cert.pem
plugins.security.ssl.http.pemkey_filepath: key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root.pem
plugins.security.allow_default_init_securityindex: true
plugins.security.nodes_dn:
  - CN=opensearch.xxx.com,OU=xxx,O=xxx,L=xxx,ST=xx,C=xx

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".op$node.max_local_storage_nodes: 3"]

pauchamb · August 15, 2022, 6:49pm

@michael.anderton
I’m not fully sure (since I haven’t tested this) if you can utilize the same certificate for different nodes. I would assume the answer is yes BUT in my cluster I have a separate cert & key for each respective node in the cluster. That bad_certificate error is usually thrown out in the case that a certificate isn’t trusted (which is resolved with plugins.security.ssl.transport.pemtrustedcas_filepath and plugins.security.ssl.http.pemtrustedcas_filepath.

Could you attempt to create a separate certificate for opensearch-data2 signed by the same root.pem certificate used to sign the first cert.pem?

pablo · August 16, 2022, 11:17am

@michael.anderton Could you check what is that remote address and which OpenSearch node is 192.168.128.4?

{localAddress=/192.168.128.4:9200, remoteAddress=/192.168.10.250:61427}

michael.anderton · August 16, 2022, 3:30pm

@pablo

The remote address is the server with the Domain Controller and CA on it (10.250) and the local address is node 1.

Topic		Replies	Views
Received fatal alert: certificate_unknown (letsencrypt, opensearch) Security security-issue	1	215	May 22, 2024
Received fatal alert: certificate_unknown - cluster Security	1	1150	July 6, 2023
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown Security configure	2	3506	May 30, 2023
CA Signed Certificates Security troubleshoot	10	1066	June 8, 2022
Certificate_unknown logged upon each successful request from Chrome or Edge Security	5	729	August 25, 2023

SSL Certificate Error "unknown_certificate"

Related topics