"Demo certificates found" error when starting Docker container without demo security configuration - even when DISABLE_INSTALL_DEMO_CONFIG is set

rtmacphail · September 20, 2021, 1:49am

Hello, I’m trying to start an OpenSearch cluster without using the demo security configuration.

I did the following (using image version 1.0.1):

Generated my own self-signed certificates as per the open search documentation.
Added new certs to volume mounts in my docker compose.
Implemented custom opensearch.yml referencing new certs.
Set plugins.security.allow_unsafe_democertificates: false
Added docker environment variable: DISABLE_INSTALL_DEMO_CONFIG=true to docker-compose.yml.

My containers fail to start with the following error:

Likely root cause: java.lang.RuntimeException: Demo certificates found [d14aefe70a592d7a29e14f3ff89c3d0070c99e87d21776aa07d333ee877e758f, 54a70016e0837a2b0c5658d1032d7ca32e432c62c55f01a2bf5adcb69a0a7ba9, bdc141ab2272c779d0f242b79063152c49e1b06a2af05e0fd90d505f2b44d5f5, 3e839e2b059036a99ee4f742814995f2fb0ced7e9d68a47851f43a3c630b5324]
        at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:367)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:726)
        at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:671)
        at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:492)
        at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:181)
        at org.opensearch.node.Node.<init>(Node.java:357)
        at org.opensearch.node.Node.<init>(Node.java:301)
        at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:240)
        at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:240)
        at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:406)
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182)
        at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:173)
        at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:99)
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:140)
        at org.opensearch.cli.Command.main(Command.java:103)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:139)
        at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:105)
For complete error details, refer to the log at /usr/share/opensearch/logs/opensearch-cluster.log

This is the file I used to generate the certs:

REM Root CA
openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -subj "/C=CA/ST=PRINCE EDWARD ISLAND/L=CHARLOTTETOWN/O=PEIGOV/OU=ITSS/CN=ROOT" -out root-ca.pem -days 730
REM Admin cert
openssl genrsa -out admin-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
openssl req -new -key admin-key.pem -subj "/C=CA/ST=PRINCE EDWARD ISLAND/L=CHARLOTTETOWN/O=PEIGOV/OU=ITSS/CN=ADMIN" -out admin.csr
openssl x509 -req -in admin.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 730
REM Node cert 1
openssl genrsa -out node-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in node-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node-key.pem
openssl req -new -key node-key.pem -subj "/C=CA/ST=PRINCE EDWARD ISLAND/L=CHARLOTTETOWN/O=PEIGOV/OU=ITSS/CN=EIPELASTIC.GOV.PE.CA" -out node.csr
openssl x509 -req -in node.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node.pem -days 730
REM Client cert
openssl genrsa -out client-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in client-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out client-key.pem
openssl req -new -key client-key.pem -subj "/C=CA/ST=PRINCE EDWARD ISLAND/L=CHARLOTTETOWN/O=PEIGOV/OU=ITSS/CN=CLIENT" -out client.csr
openssl x509 -req -in client.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out client.pem -days 730
REM Cleanup
del admin-key-temp.pem
del admin.csr
del node-key-temp.pem
del node.csr
del client-key-temp.pem
del client.csr

This is my opensearch.yml

plugins.security.ssl.transport.pemcert_filepath: node.pem
plugins.security.ssl.transport.pemkey_filepath: node-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: node.pem
plugins.security.ssl.http.pemkey_filepath: node-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - 'CN=ADMIN,OU=ITSS,O=PEIGOV,L=CHARLOTTETOWN,ST=PRINCE EDWARD ISLAND,C=CA'
plugins.security.nodes_dn:
  - 'CN=EIPELASTIC.GOV.PE.CA,OU=ITSS,O=PEIGOV,L=CHARLOTTETOWN,ST=PRINCE EDWARD ISLAND,C=CA'
plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opendistro-asynchronous-search-response*"]

This is my docker-compose.yml:

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms1G -Xmx1G" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - network.host=0.0.0.0
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - ./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./certs/node.pem:/usr/share/opensearch/config/node.pem
      - ./certs/node-key.pem:/usr/share/opensearch/config/node-key.pem
      - ./certs/admin.pem:/usr/share/opensearch/config/admin.pem
      - ./certs/admin-key.pem:/usr/share/opensearch/config/admin-key.pem
      - ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml      
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:latest
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms1G -Xmx1G"
      - network.host=0.0.0.0
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./certs/root-ca.pem:/usr/share/opensearch/config/root-ca.pem
      - ./certs/node.pem:/usr/share/opensearch/config/node.pem
      - ./certs/node-key.pem:/usr/share/opensearch/config/node-key.pem
      - ./certs/admin.pem:/usr/share/opensearch/config/admin.pem
      - ./certs/admin-key.pem:/usr/share/opensearch/config/admin-key.pem
      - ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml      
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:latest
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

I see this BUG report on the GitHub site. Wondering if this fix is in version 1.0.1.
https://github.com/opensearch-project/opensearch-build/issues/254

Thanks,
Robby.

Anthony · September 20, 2021, 5:41pm

Hi @rtmacphail The fix was merged 10 days ago and will probably be included in OpenSearch 1.1 (October 5).

If you would like to build it yourself before then, you can use below commands:

git clone https://github.com/opensearch-project/opensearch-build.git

cd opensearch-build/release/docker

./build-image-single-arch.sh -v 1.0.0 -f ./dockerfiles/opensearch.al2.dockerfile -p opensearch -a x64
./build-image-single-arch.sh -v 1.0.0 -f ./dockerfiles/opensearch-dashboards.al2.dockerfile -p opensearch-dashboards -a x64

You can then use tags 1.0.0 for opensearch and opensearch dashboards in docker-compose, with same envs previously supplied, should work as expected.

Hope this helps

rtmacphail · September 21, 2021, 12:55pm

Yes that worked. I was able to build and start the server using my self-signed certs and not the demo certs. Thanks for your help!

Topic		Replies	Views
Discovery not working on cluster with custom security OpenDistro troubleshoot	1	540	February 17, 2022
Docker-compose startup failures using v.1.2.4 first-time starts Security configure	12	1041	February 15, 2022
Replace Demo certificates Security	1	965	November 13, 2019
Replace DEMO SSL certificate for Production CA DigiCert wildcard Security troubleshoot , configure , install	10	764	May 2, 2023
How to setup security on containerised opensearch? Security configure	3	911	February 2, 2024

"Demo certificates found" error when starting Docker container without demo security configuration - even when DISABLE_INSTALL_DEMO_CONFIG is set

Related Topics