# K8s: Define SSL certificates via secrets **URL:** \ **Category:** Security\ **Tags:** configure\ **Created:** [November 4, 2021, 12:00pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523 "2021-11-04T12:00:40Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 4, 2021, 12:00pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/1 "2021-11-04T12:00:40Z")
Hello everybody, i have deployed a OpenSearch cluster on kubernetes. Everything is fine with the demo certificates, but I want to exchange them with generated SSL-certifcate. Therefore I used the _cert-manager_ to generate a certificate (which is stored in a secret) from _Let’s Encrypt_. Next I wanted to adjust my values.yaml according to the [docu](https://opensearch.org/docs/latest/security-plugin/configuration/tls/). Here I recognized, that there is no option which explains how to use certificates which are stored in secrets. Is there a workaround or am I missing something? Best regards, Simon ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 4, 2021, 12:06pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/2 "2021-11-04T12:06:04Z")
Short addition: I think OpenDistro offered something [similar](https://github.com/opendistro-for-elasticsearch/opendistro-build/blob/main/helm/README.md#ssl-1): ```auto ssl: transport: existingCertSecret: elasticsearch-transport-certs rest: enabled: true existingCertSecret: elasticsearch-rest-certs admin: enabled: true existingCertSecret: elasticsearch-admin-certs ``` ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [November 4, 2021, 2:25pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/3 "2021-11-04T14:25:24Z")
@skunz Have you tried using secretMounts? In my case I created secrets using below: ```auto apiVersion: v1 kind: Secret metadata: name: opensearch-certs namespace: default labels: app: opensearch data: node.pem: LS0tLS1CRUdJTiBDRVJUSUZJ... node-key.pem: LS0tLS1CRUdJTiBQUklWQVRFI... root-ca.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS... admin-crt.pem: LS0tLS1CRUdJTiBDRVJUSUVJQ0FUR... admin-key.pem: LS0tLS1CRUdJTiBVJUSUVJUSUVJUU... ``` Then used them in values.yaml as follows: ```auto extraVolumes: - name: opensearch-certs secret: secretName: opensearch-certs extraVolumeMounts: - name: opensearch-certs mountPath: /usr/share/opensearch/config readOnly: true ``` OR even easier by entering these into β€œsecretMounts”, as follows: ```auto secretMounts: - name: opensearch-certs secretName: opensearch-certs path: /usr/share/opensearch/config ``` ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 4, 2021, 2:32pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/4 "2021-11-04T14:32:34Z")
Thank you for the advice with secretMounts / extraVolumes! That was not on my mind πŸ˜ƒ Now the files are in the correct location and can be accessed, but they are not recognized correctly. How did you set the config of the security plugin? ```auto plugins: security: ssl: transport: pemcert_filepath: esnode.pem pemkey_filepath: esnode-key.pem pemtrustedcas_filepath: root-ca.pem enforce_hostname_verification: false http: enabled: true pemcert_filepath: esnode.pem pemkey_filepath: esnode-key.pem pemtrustedcas_filepath: root-ca.pem allow_unsafe_democertificates: true allow_default_init_securityindex: true authcz: admin_dn: - CN=kirk,OU=client,O=client,L=test, C=de audit.type: internal_opensearch enable_snapshot_restore_privilege: true check_snapshot_restore_write_privileges: true ``` ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [November 4, 2021, 2:49pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/5 "2021-11-04T14:49:33Z")
@skunz The config looks correct, however it seems you are missing the nodes\_dn section. ```auto nodes_dn: - 'CN=node1.example.com,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA' ``` Are you sure the certificates were created correctly? Did you have a look at the [this](https://opendistro.github.io/for-elasticsearch-docs/old/0.9.0/docs/security/generate-certificates/). I would recommend extracting the created certificates and mounting them into docker-compose running locally, together with updated config.yml. To ensure it works as expected. Will be easier to troubleshoot using docker-compose, prior to moving to kubernetes. I can provide a quick docker-compose to get everything running. ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 4, 2021, 4:29pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/6 "2021-11-04T16:29:08Z")
Yes, this is the example config. I thought maybe you set something additional. I will have a look into the nodes\_dn section. I am using generated certificates from β€œLet’s Encrypt”. They have a different format in the \*.crt file (server cert + 2 ca certs). So somehow I need to convert the generated file to something that’s matching the format of OpenSearch. ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [November 4, 2021, 8:54pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/7 "2021-11-04T20:54:12Z")
@skunz Yes by default the format provided by lets encrypt is different. I have just tested locally and can see that certificate provided does include 3 certs, only the first one should be used as node.crt. The second certificate is intermediate and should be used as ca (when referring to the opensearch.yml configuration). By default the key is in pkcs1 format and needs to be converted to pkcs8, I used the command below: `openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in pkcs1.key -out pkcs8.key` This can be avoided by updating the Certificate object to include below: ```auto privateKey: algorithm: RSA encoding: PKCS8 size: 2048 ``` Hope this helps ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 8, 2021, 8:52pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/8 "2021-11-08T20:52:40Z")
@Anthony Everything worked so far and the certificate is recognized and can be validated. Thank you! But I think there is a problem with the internal communication between my OpenSearch nodes. If I try to curl the cluster ```auto curl -XGET https:// -u 'admin:admin' ``` i get a β€œBad Gateway”-Message (502). Here _https://\_ points to the Ingress Controller and the connection was working before the TLS-settings. In the logs of the nodes I can see that the nodes can not connect to each other: ```auto Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record ``` I think my nodes\_dn section is wrong: ```auto ssl: transport: pemkey_filepath: certs/tls.key pemcert_filepath: certs/tls.crt pemtrustedcas_filepath: certs/tls.crt http: enabled: true pemkey_filepath: certs/tls.key pemcert_filepath: certs/tls.crt pemtrustedcas_filepath: certs/tls.crt nodes_dn: - 'CN=*.' - 'C=US,O=LetΒ΄s Encrypt,CN=R3' ``` What should be written here? And I used the certificate for both the http- and transport-Layer. Could this also be a problem? The root CA’s are also part of _tls.crt_, that’s why I used it twice. Internally in the cluster the nodes can be accessed with _opensearch-cluster-master\_. Addition: Here is the error description: ```auto opensearch [2021-11-09T08:46:12,588][WARN][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-1] handshake failed for [connectToRemoteMasterNode[172.16.0.99:9300]] opensearch org.opensearch.transport.RemoteTransportException: [opensearch-cluster-master-2][172.16.0.99:9300][internal:transport/handshake] opensearch Caused by: org.opensearch.OpenSearchException: Illegal parameter in http or transport request found. opensearch This means that one node is trying to connect to another with opensearch a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone opensearch is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/ ``` ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 9, 2021, 9:31am UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/9 "2021-11-09T09:31:54Z")
Update: ```auto nodes_dn: - 'CN=' ``` was correct. Full DN can be read with: ```auto openssl x509 -noout -in -subject ``` The errors in the logs are gone and I think it is set up correctly. ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [November 9, 2021, 11:22am UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/10 "2021-11-09T11:22:27Z")
@skunz Glad you got it working. ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 9, 2021, 12:26pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/11 "2021-11-09T12:26:40Z")
@Anthony Recently I did a clean redeploy and I get another error: ```auto [2021-11-09T12:19:55,591][WARN][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-0] handshake failed for [connectToRemoteMasterNode[172.16.0.104:9300]] opensearch org.opensearch.transport.RemoteTransportException: [opensearch-cluster-master-2][172.16.0.104:9300][internal:transport/handshake] opensearch Caused by: org.opensearch.OpenSearchException: Illegal parameter in http or transport request found. opensearch This means that one node is trying to connect to another with opensearch a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone opensearch is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/ ``` So I think something is still wrong πŸ™ˆ I will have a look into the docu and write again. Or do you have an idea on the fly? Best regards ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [November 9, 2021, 12:33pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/12 "2021-11-09T12:33:43Z")
@skunz I would check that the subject in the node certificates matches the nodes\_dn (but in reverse order). So when you run below command: `openssl x509 -noout -in -subject` Result would look like: `/DC=de/L=test/O=node/OU=node/CN=node-0.example.com` nodes\_dn entry should be the reverse: nodes\_dn: - β€˜[CN=node-0.example.com/OU=node/O=node/L=test/DC=de](http://CN=node-0.example.com/OU=node/O=node/L=test/DC=de)’ ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 9, 2021, 12:38pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/13 "2021-11-09T12:38:37Z")
@Anthony I only get a CN with my domain: ```auto subject= /CN= ``` But I use the same certificate for http- and transport-Layer. Do I need to generate a separate transport-Layer-Certificate? ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [November 9, 2021, 12:40pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/14 "2021-11-09T12:40:07Z")
No, same can be used for both ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [November 9, 2021, 12:53pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/15 "2021-11-09T12:53:18Z")
I would recommend extracting the certificates and mounting them locally into a docker-compose file, to make sure it works as expected. The file is very simple and only needs a couple of supporting files (certs, opensearch.yml and opensearch\_dashboards.yml), see example below: ```auto version: '3' services: opensearch-node1: image: opensearchproject/opensearch:1.0.0 container_name: opensearch-node1 environment: - JAVA_HOME=/usr/share/opensearch/jdk - cluster.name=opensearch-cluster - node.name=opensearch-node1 - discovery.seed_hosts=opensearch-node1,opensearch-node2 - cluster.initial_master_nodes=opensearch-node1,opensearch-node2 - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems hard: 65536 volumes: - opensearch-data1:/usr/share/opensearch/data - ./node1.key:/usr/share/opensearch/config/esnode-key.pem - ./node1.pem:/usr/share/opensearch/config/esnode.pem - ./kirk.key:/usr/share/opensearch/config/kirk-key.pem - ./kirk.pem:/usr/share/opensearch/config/kirk.pem - ./root-ca.pem:/usr/share/opensearch/config/root-ca.pem #- ./internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml #- ./roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml #- ./tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml #- ./roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml #- ./sg_action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml #- ./config.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml - ./custom-opensearch.yml:/usr/share/elasticsearch/config/opensearch.yml #- ./log4j:/usr/share/opensearch/config/log4j2.properties ports: - 9200:9200 - 9600:9600 # required for Performance Analyzer networks: - opensearch-net opensearch-node2: image: opensearchproject/opensearch:1.0.0 container_name: opensearch-node2 environment: - JAVA_HOME=/usr/share/opensearch/jdk - cluster.name=opensearch-cluster - node.name=opensearch-node2 - discovery.seed_hosts=opensearch-node1,opensearch-node2 - cluster.initial_master_nodes=opensearch-node1,opensearch-node2 - bootstrap.memory_lock=true - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 volumes: - opensearch-data2:/usr/share/opensearch/data - ./node1.key:/usr/share/opensearch/config/esnode-key.pem - ./node1.pem:/usr/share/opensearch/config/esnode.pem - ./kirk.key:/usr/share/opensearch/config/kirk-key.pem - ./kirk.pem:/usr/share/opensearch/config/kirk.pem - ./root-ca.pem:/usr/share/opensearch/config/root-ca.pem #- ./internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml #- ./roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml #- ./tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml #- ./roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml #- ./sg_action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml #- ./config.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml - ./custom-opensearch.yml:/usr/share/elasticsearch/config/opensearch.yml #- ./log4j:/usr/share/opensearch/config/log4j2.properties networks: - opensearch-net opensearch-dashboards: image: opensearchproject/opensearch-dashboards:1.0.0 container_name: opensearch-dashboards ports: - 5601:5601 expose: - "5601" volumes: - ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml #- ./spock.pem:/usr/share/opensearch-dashboards/config/spock.pem #- ./spock.key:/usr/share/opensearch-dashboards/config/spock.key environment: OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable networks: - opensearch-net volumes: opensearch-data1: opensearch-data2: networks: opensearch-net: ``` ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 9, 2021, 1:12pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/16 "2021-11-09T13:12:06Z")
But the certificate is only valid for my domain, not for my local deployment. So I can only check if the transport-Layer works. Or am I wrong? ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [November 9, 2021, 2:09pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/17 "2021-11-09T14:09:00Z")
@skunz It should work without issues if the node\_dn contains the correct entry, and plugins.security.ssl.transport.enforce\_hostname\_verification: is set to false ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 10, 2021, 8:58am UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/18 "2021-11-10T08:58:21Z")
Here is would I did so far: 1. Let’s Encrypt-certificate generated over cert-manger (lets-encrypt-production ClusterIssuers) on kubernetes. ```auto apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: opensearch-cert namespace: opensearch spec: dnsNames: - secretName: opensearch-tls privateKey: encoding: PKCS8 algorithm: RSA size: 2048 issuerRef: name: letsencrypt-prod kind: ClusterIssuer ``` - Returned a secret with two β€œfiles”: - tls.crt: 3 certificates - Domain: CN=\ - R3: CN=R3,O=Let’s Encrypt,C=US - Root: CN=ISRG Root X1,O=Internet Security Research Group,C=US - tls.key 1. Mounted secret as volume and copied both files to local folder 2. Copied current docker-compose file from OpenSearch [docu](https://opensearch.org/docs/latest/opensearch/install/docker/#sample-docker-compose-file) (I had some problems with your file, could not set env config, no idea why) and disabled install of demo certs ( **DISABLE\_INSTALL\_DEMO\_CONFIG=true** ): ```auto version: '3' services: opensearch-node1: image: opensearchproject/opensearch:1.1.0 container_name: opensearch-node1 environment: - cluster.name=opensearch-cluster - node.name=opensearch-node1 - discovery.seed_hosts=opensearch-node1,opensearch-node2 - cluster.initial_master_nodes=opensearch-node1,opensearch-node2 - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM - DISABLE_INSTALL_DEMO_CONFIG=true ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems hard: 65536 volumes: - opensearch-data1:/usr/share/opensearch/data - ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml - ./tls.crt:/usr/share/opensearch/config/tls.crt - ./tls.key:/usr/share/opensearch/config/tls.key ports: - 9200:9200 - 9600:9600 # required for Performance Analyzer networks: - opensearch-net opensearch-node2: image: opensearchproject/opensearch:1.1.0 container_name: opensearch-node2 environment: - cluster.name=opensearch-cluster - node.name=opensearch-node2 - discovery.seed_hosts=opensearch-node1,opensearch-node2 - cluster.initial_master_nodes=opensearch-node1,opensearch-node2 - bootstrap.memory_lock=true - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" - DISABLE_INSTALL_DEMO_CONFIG=true ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 volumes: - opensearch-data2:/usr/share/opensearch/data - ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml - ./tls.crt:/usr/share/opensearch/config/tls.crt - ./tls.key:/usr/share/opensearch/config/tls.key networks: - opensearch-net opensearch-dashboards: image: opensearchproject/opensearch-dashboards:1.1.0 container_name: opensearch-dashboards ports: - 5601:5601 expose: - "5601" environment: OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable networks: - opensearch-net volumes: opensearch-data1: opensearch-data2: networks: opensearch-net: ``` 1. Adjusted _custom-opensearch.yml_: ```auto plugins.security.ssl.transport.pemcert_filepath: tls.crt plugins.security.ssl.transport.pemkey_filepath: tls.key plugins.security.ssl.transport.pemtrustedcas_filepath: tls.crt plugins.security.ssl.transport.enforce_hostname_verification: false plugins.security.ssl.http.enabled: true plugins.security.ssl.http.pemcert_filepath: tls.crt plugins.security.ssl.http.pemkey_filepath: tls.key plugins.security.ssl.http.pemtrustedcas_filepath: tls.crt plugins.security.allow_unsafe_democertificates: false plugins.security.allow_default_init_securityindex: true plugins.security.authcz.admin_dn: - CN= plugins.security.nodes_dn: - 'CN=' ``` 1. Run _docker-compose up_ and both nodes came up but could not find each other: ```auto opensearch-node2 | [2021-11-10T08:49:20,482][WARN][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [opensearch-node1, opensearch-node2] to bootstrap a cluster: have discovered [{opensearch-node2}{k95-Ww5XTR-JhxPvMq4VaA}{xHmNDcjVSsebgZYogZjRIg}{127.0.0.1}{127.0.0.1:9300}{dimr}]; discovery will continue using [192.168.0.3:9300, 192.168.0.4:9300] from hosts providers and [{opensearch-node2}{k95-Ww5XTR-JhxPvMq4VaA}{xHmNDcjVSsebgZYogZjRIg}{127.0.0.1}{127.0.0.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 opensearch-node1 | [2021-11-10T08:49:20,606][WARN][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [opensearch-node1, opensearch-node2] to bootstrap a cluster: have discovered [{opensearch-node1}{1IcawTxOTEqJf4EOSw5mYQ}{T1hR1L45QMapNQYKo1RldQ}{127.0.0.1}{127.0.0.1:9300}{dimr}]; discovery will continue using [192.168.0.3:9300, 192.168.0.4:9300] from hosts providers and [{opensearch-node1}{1IcawTxOTEqJf4EOSw5mYQ}{T1hR1L45QMapNQYKo1RldQ}{127.0.0.1}{127.0.0.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0 ``` So I think something is still wrong with the way the certificate(s) are integrated. **Questions:** - Is it important to set the files _Kirk.key_ and _Kirk.pem_ and are they needed? - Since all three certificates are in one file, do they need to be extracted somehow? E.g. a extra file only with the root cert? - Is my _nodes\_dn_-section still wrong? ---
**Author:** ![Anthony](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/anthony/32/9939_2.png) [@Anthony](https://forum.opensearch.org/u/Anthony)\ **Post date:** [November 10, 2021, 11:39am UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/19 "2021-11-10T11:39:43Z")
@skunz Please add line below to custom-opensearch.yml `network.host: 0.0.0.0` I’ve just tested using 3 certs in one, and it all works as expected. So no need to extract any certs, (I used your config exactly). To answer rest of the questions: Regarding admin cert (kirk), it is not needed, however if you are going to be making changes via securityadmin.sh script (including retrieving config from security index) then this will be required. In general it is recommended to have one. _nodes\_dn_-section seems correct. ---
**Author:** ![skunz](https://avatars.discourse-cdn.com/v4/letter/s/958977/32.png) [@skunz](https://forum.opensearch.org/u/skunz)\ **Post date:** [November 10, 2021, 2:49pm UTC](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523/20 "2021-11-10T14:49:07Z")
@Anthony ```auto network.host: 0.0.0.0 ``` fixed the docker-compose problem, thank you! But this line is not needed in case of the k8s deployment or is it? I moved this config (without the line above) to k8s and the nodes came up and were connected. Chart.yml ```auto # helm install -n opensearch --create-namespace opensearch-with-dashboard . --values values.yaml apiVersion: v2 name: opensearch-with-dashboard ### IMPORTANT: Bump this version every time you change anything in this chart (e.g. dependencies, versions, templates, files, etc.) version: 0.0.0 dependencies: - name: opensearch ### IMPORTANT: Also run "helm dependency update ." in this directory if you update any dependency version. version: 1.2.4 repository: "https://opensearch-project.github.io/helm-charts/" - name: opensearch-dashboards ### IMPORTANT: Also run "helm dependency update ." in this directory if you update any dependency version. version: 1.0.6 repository: "https://opensearch-project.github.io/helm-charts/" ``` values.yml ```auto opensearch: config: opensearch.yml: plugins.security.ssl.transport.pemcert_filepath: certs/tls.crt plugins.security.ssl.transport.pemkey_filepath: certs/tls.key plugins.security.ssl.transport.pemtrustedcas_filepath: certs/tls.crt plugins.security.ssl.transport.enforce_hostname_verification: false plugins.security.ssl.http.enabled: true plugins.security.ssl.http.pemcert_filepath: certs/tls.crt plugins.security.ssl.http.pemkey_filepath: certs/tls.key plugins.security.ssl.http.pemtrustedcas_filepath: certs/tls.crt plugins.security.allow_unsafe_democertificates: false plugins.security.authcz.admin_dn: - CN= plugins.security.nodes_dn: - 'CN=' persistence: enabled: true # Set to false to disable the `fsgroup-volume` initContainer that will update permissions on the persistent disk. enableInitChown: true labels: # Add default labels for the volumeClaimTemplate of the StatefulSet enabled: false # OpenSearch Persistent Volume Storage Class # If defined, storageClassName: # If set to "-", storageClassName: "", which disables dynamic provisioning # If undefined (the default) or set to null, no storageClassName spec is # set, choosing the default provisioner. (gp2 on AWS, standard on # GKE, AWS & OpenStack) # storageClass: csi-disk accessModes: - ReadWriteOnce size: 25Gi annotations: everest.io/disk-volume-type: SSD # Enabling this will publically expose your OpenSearch instance. # Only enable this if you have security enabled on your cluster ingress: enabled: true annotations: kubernetes.io/ingress.class: nginx-external # kubernetes.io/ingress.class: nginx # kubernetes.io/tls-acme: "true" path: / hosts: - tls: - secretName: opensearch-tls extraVolumes: - name: certificate secret: secretName: opensearch-tls extraVolumeMounts: - name: certificate mountPath: "/usr/share/opensearch/config/certs" readOnly: true extraEnvs: - name: DISABLE_INSTALL_DEMO_CONFIG value: "true" opensearch-dashboards: # Cluster-internal connection opensearchHosts: "https://opensearch-cluster-master:9200" imageTag: "" #config: # Default OpenSearch Dashboards configuration from docker image of Dashboards # opensearch_dashboards.yml: # opensearch.ssl.verificationMode: none # server.ssl.enabled: false ingress: enabled: true annotations: kubernetes.io/ingress.class: nginx-external #kubernetes.io/tls-acme: "true" hosts: - host: paths: - path: / backend: serviceName: opensearch-dashboards-dev servicePort: 5601 tls: - secretName: opensearch-dashboard-cert ``` Both ingress rules deliver the right certificate. If I connect over Dashboards to OpenSearch everything is ok (makes sense since verification disabled). If I connect over my Browser (Firefox) I can see that the certificate is correct and valid but I get **502 Bad Gateway** from Nginx. in the _ingress-nginx-external-controller_ for the ingress from OpenSearch I get the following errors: ```auto 2021/11/10 14:41:38 [error[] 4261#4261: *32847247 upstream prematurely closed connection while reading response header from upstream, client: 100.125.67.47, server: , request: "GET / HTTP/2.0", upstream: "http://172.16.0.61:9200/", host: "" 2021/11/10 14:41:38 [error[] 4261#4261: *32847247 upstream prematurely closed connection while reading response header from upstream, client: 100.125.67.47, server: , request: "GET / HTTP/2.0", upstream: "http://172.16.0.108:9200/", host: "" 2021/11/10 14:41:38 [error[] 4261#4261: *32847247 upstream prematurely closed connection while reading response header from upstream, client: 100.125.67.47, server: , request: "GET / HTTP/2.0", upstream: "http://172.16.0.139:9200/", host: "" ``` Here are the OpenSearch pods: ```auto β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ Pods(opensearch)[4] ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ β”‚ NAME PF READY RESTARTS STATUS CPU MEM↓ %CPU/R %CPU/L %MEM/R %MEM/L IP NODE AGE β”‚ β”‚ opensearch-cluster-master-1 ● 1/1 0 Running 16 1246 1 n/a 1246 n/a 172.16.0.61 10.202.102.43 74m β”‚ β”‚ opensearch-cluster-master-2 ● 1/1 0 Running 28 1245 2 n/a 1245 n/a 172.16.0.108 10.202.102.36 75m β”‚ β”‚ opensearch-cluster-master-0 ● 1/1 0 Running 25 1131 2 n/a 1131 n/a 172.16.0.139 10.202.102.159 74m β”‚ ``` Every OpenSearch-Pod has more or less the same error message: ```auto opensearch [2021-11-10T14:51:26,174[][ERROR[][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport[] [opensearch-cluster-master-2[] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310... (shorted) opensearch io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310... (shorted) opensearch at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1261) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final[] opensearch at java.lang.Thread.run(Thread.java:832) [?:?] opensearch [2021-11-10T14:51:26,175[][WARN [][o.o.h.AbstractHttpServerTransport[] [opensearch-cluster-master-2[] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.16.0.108:9200, remoteAddress=/172.16.0.52:46764} opensearch io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310... (shorted) opensearch at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final[] opensearch at java.lang.Thread.run(Thread.java:832) [?:?] opensearch Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310... (shorted) opensearch at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1261) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[] opensearch at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[] opensearch ... 16 more ``` The decoded shorted message is hex format and is an HTTP-Request: ```auto GET / HTTP/1.1 Host: X-Request-ID: 9c50cec41af24277b5e869f99e657733 X-Real-IP: 100.125.67.47 X-Forwarded-For: 100.125.67.47 X-Forwarded-Host: X-Forwarded-Port: 443 X-Forwarded-Proto: https X-Scheme: https user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0 accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8 accept-language: de,en-US;q=0.7,en;q=0.3 accept-encoding: gzip, deflate, br authorization: Basic YWRtaW46YWRtaW4= upgrade-insecure-requests: 1 sec-fetch-dest: document sec-fetch-mode: navigate sec-fetch-site: none sec-fetch-user: ?1 cache-control: max-age=0 ``` Has this something to do with timeouts or is still something misconfigured? I think I forget to mention that the kubernetes cluster is only accessible through an **VPN-Tunnel**. But the **\** is accessible from everywhere. Could this also be part of the problem? [Next page](https://forum.opensearch.org/t/k8s-define-ssl-certificates-via-secrets/7523.md?page=2)