Enable TLS for Remote Access to OpenSearch Dashboards and Communication Between OpenSearch and OpenSearch Dashboards

HotDog2217 · February 25, 2025, 1:05pm

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OS of the server running the container:
Ubuntu 24.04 LTS

Docker:
docker-ce/noble,now 5:27.5.1-1~ubuntu.24.04~noble amd64
docker-compose-plugin/noble,now 2.32.4-1~ubuntu.24.04~noble amd64

Docker images:
opensearchproject/opensearch:latest
opensearchproject/opensearch-dashboards:latest

Describe the issue:
I created a self-signed certificate and docker-compose.yml following the official documentation.
After starting each container, when accessing https://<Host_IP_Address>:5601 remotely, the page displays:
“OpenSearch Dashboards server is not ready yet”, and the OpenSearch Dashboards screen does not load.

Configuration:
Regarding the self-signed certificate, I followed the steps from “Generate a private key” to “Generate an admin certificate” on the following site to generate admin.pem, admin-key.pem, and root-ca.pem.
https://opensearch.org/docs/latest/security/configuration/generate-certificates/

For the docker-compose.yml file, I referred to the configuration values at the bottom of the following page:
https://opensearch.org/docs/latest/install-and-configure/install-dashboards/tls/

The specific content of docker-compose.yml is attached below.

services:
  opensearch:
    image: opensearchproject/opensearch:latest
    hostname: opensearch-node1
    container_name: opensearch-node1
    environment:
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - bootstrap.memory_lock=true
      - "DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI=true"
      - cluster.routing.allocation.disk.threshold_enabled=false
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - node.store.allow_mmap=false
      - network.host=0.0.0.0
      - discovery.seed_hosts=opensearch-node1
      - cluster.initial_cluster_manager_nodes=opensearch-node1
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-node1-data:/usr/share/opensearch/data
    ports:
      - 9200:9200
    networks:
      - opensearch-net
    restart: unless-stopped
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    environment:
      SERVER_HOST: '0.0.0.0'
      SERVER_SSL_ENABLED: true
      SERVER_SSL_CERTIFICATE: /usr/share/opensearch-dashboards/config/admin.pem
      SERVER_SSL_KEY: /usr/share/opensearch-dashboards/config/admin-key.pem
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200"]'
      OPENSEARCH_SSL_VERIFICATIONMODE: full
      OPENSEARCH_SSL_CERTIFICATEAUTHORITIES: '["/usr/share/opensearch-dashboards/config/root-ca.pem"]'
      OPENSEARCH_USERNAME: "kibanaserver"
      OPENSEARCH_PASSWORD: "kibanaserver"
      OPENSEARCH_REQUESTHEADERSALLOWLIST: '[ authorization,securitytenant ]'
      OPENSEARCH_SECURITY_MULTITENANCY_ENABLED: true
      OPENSEARCH_SECURITY_MULTITENANCY_TENANTS_PREFERRED: '["Private", "Global"]'
      OPENSEARCH_SECURITY_READONLY_MODE_ROLES: '["kibana_read_only"]'
      OPENSEARCH_SECURITY_COOKIE_SECURE: true
    ports:
      - 5601:5601
    expose:
      - '5601'
    volumes:
      - ./config/admin.pem:/usr/share/opensearch-dashboards/config/admin.pem
      - ./config/admin-key.pem:/usr/share/opensearch-dashboards/config/admin-key.pem
      - ./config/root-ca.pem:/usr/share/opensearch-dashboards/config/root-ca.pem
    networks:
      - opensearch-net

volumes:
  opensearch-node1-data:

networks:
  opensearch-net:

Relevant Logs or Screenshots:
After starting each container, I ran the command $ docker compose logs opensearch-dashboards, and the output was as follows:

opensearch-dashboards | {“type”:“log”,“@timestamp”:“2025-02-25T09:34:51Z”,“tags”:[“error”,“opensearch”,“data”],“pid”:1,“message”:“[ConnectionError]: unable to verify the first certificate”}

Eugene7 · February 25, 2025, 4:31pm

Hi @HotDog2217

In the opensearch.ssl.certificateAuthorities parameter, do you use a demo certificate, or is it your custom-generated one?

Eugene7 · February 25, 2025, 4:34pm

According to your current docker-compose.yml file, you are using demo certificates for the OpenSearch cluster.

HotDog2217 · February 26, 2025, 1:46am

Hi @Eugene7
Thank you for replying to this topic.

The value set in the opensearch.ssl.certificateAuthorities parameter (/usr/share/opensearch-dashboards/config/root-ca.pem) is a custom-generated certificate.

I created the certificate by following the steps outlined in the documentation below:
https://opensearch.org/docs/latest/security/configuration/generate-certificates/#generate-a-root-certificate

After that, I configured the volumes parameter in the opensearch-dashboards section of the docker-compose.yml file to pass the certificate from the host to the container, as shown below:

    volumes:
      - ./config/admin.pem:/usr/share/opensearch-dashboards/config/admin.pem
      - ./config/admin-key.pem:/usr/share/opensearch-dashboards/config/admin-key.pem
      - ./config/root-ca.pem:/usr/share/opensearch-dashboards/config/root-ca.pem

If you need any additional information, please let me know!

Eugene7 · February 26, 2025, 6:13pm

According to the documentation , opensearch.ssl.certificateAuthorities parameter has to have the root CA certificate of the OpenSearch cluster. If you are using demo certificates in your OpenSearch cluster, you have to use the same demo root CA in opensearch.ssl.certificateAuthorities.

HotDog2217 · February 27, 2025, 12:14pm

I understand that if you are using demo certificates in the OpenSearch cluster, you also need to specify the demo root CA in opensearch.ssl.certificateAuthorities. However, in my environment, I intend to use custom certificates that I created myself instead of demo certificates. Is there a possibility that there is a configuration mistake causing demo certificates to be used?

Eugene7 · February 28, 2025, 3:40pm

Hi @HotDog2217 ,

To use your certificates in the OpenSearch cluster instead of demo certificates, you must add them to the OpenSearch container. To do that, you must also create node certificates and mount them to the OpenSearch container.

HotDog2217 · March 3, 2025, 3:22pm

Hi @Eugene7

I apologize for the delayed response, and I truly appreciate your continued assistance with my issue.
Based on the information you provided, I have modified the “docker-compose.yml” file. After starting the containers, I attempted to access https://<Host_IP_Address>:5601 remotely, but the page displayed “OpenSearch Dashboards server is not ready yet,” and I was unable to load the OpenSearch Dashboards screen. There has been no change from the situation when I initially asked the question. Could you please help me understand what might be wrong? The contents of the “docker-compose.yml” file are as follows:

services:
  opensearch:
    image: opensearchproject/opensearch:latest
    hostname: opensearch-node1
    container_name: opensearch-node1
    environment:
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - bootstrap.memory_lock=true
      - "DISABLE_PERFORMANCE_ANALYZER_AGENT_CLI=true"
      - cluster.routing.allocation.disk.threshold_enabled=false
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - node.store.allow_mmap=false
      - network.host=0.0.0.0
      - discovery.seed_hosts=opensearch-node1
      - cluster.initial_cluster_manager_nodes=opensearch-node1
      - OPENSEARCH_INITIAL_ADMIN_PASSWORD=${OPENSEARCH_INITIAL_ADMIN_PASSWORD}
      - plugins.security.ssl.transport.pemcert_filepath=node1.pem
      - plugins.security.ssl.transport.pemkey_filepath=node1-key.pem
      - plugins.security.ssl.transport.pemtrustedcas_filepath=root-ca.pem
      - plugins.security.ssl.transport.enforce_hostname_verification=false
      - plugins.security.ssl.http.enabled=true
      - plugins.security.ssl.http.pemcert_filepath=node1.pem
      - plugins.security.ssl.http.pemkey_filepath=node1-key.pem
      - plugins.security.ssl.http.pemtrustedcas_filepath=root-ca.pem
      - plugins.security.authcz.admin_dn='CN=A,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
      - plugins.security.nodes_dn='CN=<Host_IP_Address>,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-node1-data:/usr/share/opensearch/data
      - ./config/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./config/node1.pem:/usr/share/opensearch/config/node1.pem
      - ./config/node1-key.pem:/usr/share/opensearch/config/node1-key.pem
    ports:
      - 9200:9200
    networks:
      - opensearch-net
    restart: unless-stopped
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    environment:
      SERVER_HOST: '0.0.0.0'
      SERVER_SSL_ENABLED: true
      SERVER_SSL_CERTIFICATE: /usr/share/opensearch-dashboards/config/node1.pem
      SERVER_SSL_KEY: /usr/share/opensearch-dashboards/config/node1-key.pem
      OPENSEARCH_HOSTS: '[https://opensearch-node1:9200]'
      OPENSEARCH_SSL_VERIFICATIONMODE: full
      OPENSEARCH_SSL_CERTIFICATEAUTHORITIES: '["/usr/share/opensearch-dashboards/config/root-ca.pem"]'
      OPENSEARCH_USERNAME: "kibanaserver"
      OPENSEARCH_PASSWORD: "kibanaserver"
      OPENSEARCH_REQUESTHEADERSALLOWLIST: '[ authorization,securitytenant ]'
      OPENSEARCH_SECURITY_MULTITENANCY_ENABLED: true
      OPENSEARCH_SECURITY_MULTITENANCY_TENANTS_PREFERRED: '["Private", "Global"]'
      OPENSEARCH_SECURITY_READONLY_MODE_ROLES: '["kibana_read_only"]'
      OPENSEARCH_SECURITY_COOKIE_SECURE: true
    ports:
      - 5601:5601
    expose:
      - '5601'
    volumes:
      - ./config/root-ca.pem:/usr/share/opensearch-dashboards/config/root-ca.pem
      - ./config/node1.pem:/usr/share/opensearch-dashboards/config/node1.pem
      - ./config/node1-key.pem:/usr/share/opensearch-dashboards/config/node1-key.pem
    networks:
      - opensearch-net

volumes:
  opensearch-node1-data:

networks:
  opensearch-net:

I created the “node1.pem” and “node1-key.pem” certificates based on the URL you provided.
https://opensearch.org/docs/2.19/security/configuration/generate-certificates/#optional-generate-node-and-client-certificates

Eugene7 · March 11, 2025, 3:54pm

Could you please try changing opensearch.ssl.verificationMode from full to certificate?

Eugene7 · March 12, 2025, 2:41pm

Do you see any error messages in the OpenSearch or OpenSearch Dashboards logs?

Topic		Replies	Views
Opensearch + Opensearch dashboard Connection error Security troubleshoot	3	5629	December 29, 2022
Issue with OpenID Connect (OIDC) and TLS Security security-issue	6	906	April 4, 2023
Issue with OpenSearch dashboard TLS Configuration Security troubleshoot	14	2135	June 1, 2023
NotSslRecordException: not an SSL/TLS record Security discuss , troubleshoot , configure	20	2351	July 15, 2024
Error self-signed certificate Security troubleshoot	9	441	July 11, 2024

Enable TLS for Remote Access to OpenSearch Dashboards and Communication Between OpenSearch and OpenSearch Dashboards

Related topics