K8s: Define SSL certificates via secrets

skunz · November 4, 2021, 12:00pm

Hello everybody,

i have deployed a OpenSearch cluster on kubernetes.
Everything is fine with the demo certificates, but I want to exchange them with generated SSL-certifcate.
Therefore I used the cert-manager to generate a certificate (which is stored in a secret) from Let’s Encrypt.
Next I wanted to adjust my values.yaml according to the docu.

Here I recognized, that there is no option which explains how to use certificates which are stored in secrets.
Is there a workaround or am I missing something?

Best regards,
Simon

skunz · November 4, 2021, 12:06pm

Short addition:
I think OpenDistro offered something similar:

ssl:
  transport:
    existingCertSecret: elasticsearch-transport-certs
  rest:
    enabled: true
    existingCertSecret: elasticsearch-rest-certs
  admin:
    enabled: true
    existingCertSecret: elasticsearch-admin-certs

Anthony · November 4, 2021, 2:25pm

@skunz
Have you tried using secretMounts?

In my case I created secrets using below:

apiVersion: v1
kind: Secret
metadata:
  name: opensearch-certs
  namespace: default
  labels:
    app: opensearch
data:
  node.pem: LS0tLS1CRUdJTiBDRVJUSUZJ...
  node-key.pem: LS0tLS1CRUdJTiBQUklWQVRFI...
  root-ca.pem: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS...
  admin-crt.pem: LS0tLS1CRUdJTiBDRVJUSUVJQ0FUR...
  admin-key.pem: LS0tLS1CRUdJTiBVJUSUVJUSUVJUU...

Then used them in values.yaml as follows:

extraVolumes:
   - name: opensearch-certs
     secret:
       secretName: opensearch-certs

extraVolumeMounts:
   - name: opensearch-certs
     mountPath: /usr/share/opensearch/config
     readOnly: true

OR even easier by entering these into “secretMounts”, as follows:

secretMounts:
  - name: opensearch-certs
    secretName: opensearch-certs
    path: /usr/share/opensearch/config

skunz · November 4, 2021, 2:32pm

Thank you for the advice with secretMounts / extraVolumes! That was not on my mind

Now the files are in the correct location and can be accessed, but they are not recognized correctly.
How did you set the config of the security plugin?

plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: esnode.pem
            pemkey_filepath: esnode-key.pem
            pemtrustedcas_filepath: root-ca.pem
        allow_unsafe_democertificates: true
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=kirk,OU=client,O=client,L=test, C=de
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true

Anthony · November 4, 2021, 2:49pm

@skunz
The config looks correct, however it seems you are missing the nodes_dn section.

nodes_dn:
  - 'CN=node1.example.com,OU=UNIT,O=ORG,L=TORONTO,ST=ONTARIO,C=CA'

Are you sure the certificates were created correctly? Did you have a look at the this.

I would recommend extracting the created certificates and mounting them into docker-compose running locally, together with updated config.yml. To ensure it works as expected. Will be easier to troubleshoot using docker-compose, prior to moving to kubernetes. I can provide a quick docker-compose to get everything running.

skunz · November 4, 2021, 4:29pm

Yes, this is the example config. I thought maybe you set something additional.
I will have a look into the nodes_dn section.

I am using generated certificates from “Let’s Encrypt”.
They have a different format in the *.crt file (server cert + 2 ca certs).

So somehow I need to convert the generated file to something that’s matching the format of OpenSearch.

Anthony · November 4, 2021, 8:54pm

@skunz
Yes by default the format provided by lets encrypt is different.

I have just tested locally and can see that certificate provided does include 3 certs, only the first one should be used as node.crt. The second certificate is intermediate and should be used as ca (when referring to the opensearch.yml configuration).
By default the key is in pkcs1 format and needs to be converted to pkcs8, I used the command below:

openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in pkcs1.key -out pkcs8.key

This can be avoided by updating the Certificate object to include below:

privateKey:
    algorithm: RSA
    encoding: PKCS8
    size: 2048

Hope this helps

skunz · November 8, 2021, 8:52pm

@Anthony

Everything worked so far and the certificate is recognized and can be validated.
Thank you!

But I think there is a problem with the internal communication between my OpenSearch nodes.
If I try to curl the cluster

curl -XGET https://<domain> -u 'admin:admin'

i get a “Bad Gateway”-Message (502).
Here https://<domain> points to the Ingress Controller and the connection was working before the TLS-settings.
In the logs of the nodes I can see that the nodes can not connect to each other:

Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record

I think my nodes_dn section is wrong:

          ssl:
            transport:
              pemkey_filepath: certs/tls.key
              pemcert_filepath: certs/tls.crt
              pemtrustedcas_filepath: certs/tls.crt
            http:
              enabled: true
              pemkey_filepath: certs/tls.key
              pemcert_filepath: certs/tls.crt
              pemtrustedcas_filepath: certs/tls.crt
          nodes_dn:
            - 'CN=*.<domain>'
            - 'C=US,O=Let´s Encrypt,CN=R3'

What should be written here?
And I used the certificate for both the http- and transport-Layer. Could this also be a problem?
The root CA’s are also part of tls.crt, that’s why I used it twice.
Internally in the cluster the nodes can be accessed with opensearch-cluster-master<id>.

Addition:
Here is the error description:

opensearch [2021-11-09T08:46:12,588][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-1] handshake failed for [connectToRemoteMasterNode[172.16.0.99:9300]]                                                                  
opensearch org.opensearch.transport.RemoteTransportException: [opensearch-cluster-master-2][172.16.0.99:9300][internal:transport/handshake]                                                                                                              
opensearch Caused by: org.opensearch.OpenSearchException: Illegal parameter in http or transport request found.                                                                                                                                            
opensearch This means that one node is trying to connect to another with                                                                                                                                                                                 
opensearch a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone                                                                                                                                                      
opensearch is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/

skunz · November 9, 2021, 9:31am

Update:

nodes_dn:
            - 'CN=<domain>'

was correct.

Full DN can be read with:

openssl x509 -noout -in <certificate file name with full path> -subject

The errors in the logs are gone and I think it is set up correctly.

Anthony · November 9, 2021, 11:22am

@skunz Glad you got it working.

skunz · November 9, 2021, 12:26pm

@Anthony

Recently I did a clean redeploy and I get another error:

[2021-11-09T12:19:55,591][WARN ][o.o.d.HandshakingTransportAddressConnector] [opensearch-cluster-master-0] handshake failed for [connectToRemoteMasterNode[172.16.0.104:9300]]                                                                 
opensearch org.opensearch.transport.RemoteTransportException: [opensearch-cluster-master-2][172.16.0.104:9300][internal:transport/handshake]                                                                                                              
opensearch Caused by: org.opensearch.OpenSearchException: Illegal parameter in http or transport request found.                                                                                                                                           
opensearch This means that one node is trying to connect to another with                                                                                                                                                                                   
opensearch a non-node certificate (no OID or security.nodes_dn incorrect configured) or that someone                                                                                                                                                       
opensearch is spoofing requests. Check your TLS certificate setup as described here: See https://opendistro.github.io/for-elasticsearch-docs/docs/troubleshoot/tls/

So I think something is still wrong
I will have a look into the docu and write again.
Or do you have an idea on the fly?

Best regards

Anthony · November 9, 2021, 12:33pm

@skunz I would check that the subject in the node certificates matches the nodes_dn (but in reverse order).

So when you run below command:

openssl x509 -noout -in <certificate file name with full path> -subject

Result would look like:

/DC=de/L=test/O=node/OU=node/CN=node-0.example.com

nodes_dn entry should be the reverse:

nodes_dn:
- ‘CN=node-0.example.com/OU=node/O=node/L=test/DC=de’

skunz · November 9, 2021, 12:38pm

@Anthony

I only get a CN with my domain:

subject= /CN=<domain>

But I use the same certificate for http- and transport-Layer.
Do I need to generate a separate transport-Layer-Certificate?

Anthony · November 9, 2021, 12:40pm

No, same can be used for both

Anthony · November 9, 2021, 12:53pm

I would recommend extracting the certificates and mounting them locally into a docker-compose file, to make sure it works as expected. The file is very simple and only needs a couple of supporting files (certs, opensearch.yml and opensearch_dashboards.yml), see example below:

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:1.0.0
    container_name: opensearch-node1
    environment:
      - JAVA_HOME=/usr/share/opensearch/jdk
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - ./node1.key:/usr/share/opensearch/config/esnode-key.pem
      - ./node1.pem:/usr/share/opensearch/config/esnode.pem
      - ./kirk.key:/usr/share/opensearch/config/kirk-key.pem
      - ./kirk.pem:/usr/share/opensearch/config/kirk.pem
      - ./root-ca.pem:/usr/share/opensearch/config/root-ca.pem

      #- ./internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
      #- ./roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml
      #- ./tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml
      #- ./roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml
      #- ./sg_action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml
      #- ./config.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
      - ./custom-opensearch.yml:/usr/share/elasticsearch/config/opensearch.yml
      #- ./log4j:/usr/share/opensearch/config/log4j2.properties
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:1.0.0
    container_name: opensearch-node2
    environment:
      - JAVA_HOME=/usr/share/opensearch/jdk
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./node1.key:/usr/share/opensearch/config/esnode-key.pem
      - ./node1.pem:/usr/share/opensearch/config/esnode.pem
      - ./kirk.key:/usr/share/opensearch/config/kirk-key.pem
      - ./kirk.pem:/usr/share/opensearch/config/kirk.pem
      - ./root-ca.pem:/usr/share/opensearch/config/root-ca.pem

      
      #- ./internal_users.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
      #- ./roles_mapping.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml
      #- ./tenants.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml
      #- ./roles.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml
      #- ./sg_action_groups.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml
      #- ./config.yml:/usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
      - ./custom-opensearch.yml:/usr/share/elasticsearch/config/opensearch.yml
      #- ./log4j:/usr/share/opensearch/config/log4j2.properties
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:1.0.0
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    volumes:
      - ./opensearch_dashboards.yml:/usr/share/opensearch-dashboards/config/opensearch_dashboards.yml
      #- ./spock.pem:/usr/share/opensearch-dashboards/config/spock.pem
      #- ./spock.key:/usr/share/opensearch-dashboards/config/spock.key
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

skunz · November 9, 2021, 1:12pm

But the certificate is only valid for my domain, not for my local deployment.
So I can only check if the transport-Layer works.
Or am I wrong?

Anthony · November 9, 2021, 2:09pm

@skunz It should work without issues if the node_dn contains the correct entry, and
plugins.security.ssl.transport.enforce_hostname_verification: is set to false

skunz · November 10, 2021, 8:58am

Here is would I did so far:

Let’s Encrypt-certificate generated over cert-manger (lets-encrypt-production ClusterIssuers) on kubernetes.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: opensearch-cert
  namespace: opensearch
spec:
  dnsNames:
  - <domain>
  secretName: opensearch-tls
  privateKey:
    encoding: PKCS8
    algorithm: RSA
    size: 2048
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer

Returned a secret with two “files”:
- tls.crt: 3 certificates
  - Domain: CN=<domain>
  - R3: CN=R3,O=Let’s Encrypt,C=US
  - Root: CN=ISRG Root X1,O=Internet Security Research Group,C=US
- tls.key

Mounted secret as volume and copied both files to local folder
Copied current docker-compose file from OpenSearch docu (I had some problems with your file, could not set env config, no idea why) and disabled install of demo certs (DISABLE_INSTALL_DEMO_CONFIG=true):

version: '3'
services:
  opensearch-node1:
    image: opensearchproject/opensearch:1.1.0
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node1
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
      - DISABLE_INSTALL_DEMO_CONFIG=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data
      - ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./tls.crt:/usr/share/opensearch/config/tls.crt
      - ./tls.key:/usr/share/opensearch/config/tls.key
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - opensearch-net
  opensearch-node2:
    image: opensearchproject/opensearch:1.1.0
    container_name: opensearch-node2
    environment:
      - cluster.name=opensearch-cluster
      - node.name=opensearch-node2
      - discovery.seed_hosts=opensearch-node1,opensearch-node2
      - cluster.initial_master_nodes=opensearch-node1,opensearch-node2
      - bootstrap.memory_lock=true
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
      - DISABLE_INSTALL_DEMO_CONFIG=true
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    volumes:
      - opensearch-data2:/usr/share/opensearch/data
      - ./custom-opensearch.yml:/usr/share/opensearch/config/opensearch.yml
      - ./tls.crt:/usr/share/opensearch/config/tls.crt
      - ./tls.key:/usr/share/opensearch/config/tls.key
    networks:
      - opensearch-net
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:1.1.0
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable
    networks:
      - opensearch-net

volumes:
  opensearch-data1:
  opensearch-data2:

networks:
  opensearch-net:

Adjusted custom-opensearch.yml:

plugins.security.ssl.transport.pemcert_filepath: tls.crt
plugins.security.ssl.transport.pemkey_filepath: tls.key
plugins.security.ssl.transport.pemtrustedcas_filepath: tls.crt
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: tls.crt
plugins.security.ssl.http.pemkey_filepath: tls.key
plugins.security.ssl.http.pemtrustedcas_filepath: tls.crt
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:
  - CN=<domain>
plugins.security.nodes_dn:
  - 'CN=<domain>'

Run docker-compose up and both nodes came up but could not find each other:

opensearch-node2         | [2021-11-10T08:49:20,482][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [opensearch-node1, opensearch-node2] to bootstrap a cluster: have discovered [{opensearch-node2}{k95-Ww5XTR-JhxPvMq4VaA}{xHmNDcjVSsebgZYogZjRIg}{127.0.0.1}{127.0.0.1:9300}{dimr}]; discovery will continue using [192.168.0.3:9300, 192.168.0.4:9300] from hosts providers and [{opensearch-node2}{k95-Ww5XTR-JhxPvMq4VaA}{xHmNDcjVSsebgZYogZjRIg}{127.0.0.1}{127.0.0.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
opensearch-node1         | [2021-11-10T08:49:20,606][WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-node1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [opensearch-node1, opensearch-node2] to bootstrap a cluster: have discovered [{opensearch-node1}{1IcawTxOTEqJf4EOSw5mYQ}{T1hR1L45QMapNQYKo1RldQ}{127.0.0.1}{127.0.0.1:9300}{dimr}]; discovery will continue using [192.168.0.3:9300, 192.168.0.4:9300] from hosts providers and [{opensearch-node1}{1IcawTxOTEqJf4EOSw5mYQ}{T1hR1L45QMapNQYKo1RldQ}{127.0.0.1}{127.0.0.1:9300}{dimr}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

So I think something is still wrong with the way the certificate(s) are integrated.

Questions:

Is it important to set the files Kirk.key and Kirk.pem and are they needed?
Since all three certificates are in one file, do they need to be extracted somehow? E.g. a extra file only with the root cert?
Is my nodes_dn-section still wrong?

Anthony · November 10, 2021, 11:39am

@skunz
Please add line below to custom-opensearch.yml

network.host: 0.0.0.0

I’ve just tested using 3 certs in one, and it all works as expected. So no need to extract any certs, (I used your config exactly).
To answer rest of the questions:

Regarding admin cert (kirk), it is not needed, however if you are going to be making changes via securityadmin.sh script (including retrieving config from security index) then this will be required. In general it is recommended to have one.

nodes_dn-section seems correct.

skunz · November 10, 2021, 2:49pm

@Anthony

network.host: 0.0.0.0

fixed the docker-compose problem, thank you!
But this line is not needed in case of the k8s deployment or is it?

I moved this config (without the line above) to k8s and the nodes came up and were connected.

Chart.yml

# helm install -n opensearch --create-namespace opensearch-with-dashboard . --values values.yaml

apiVersion: v2
name: opensearch-with-dashboard
### IMPORTANT: Bump this version every time you change anything in this chart (e.g. dependencies, versions, templates, files, etc.)
version: 0.0.0
dependencies:
- name: opensearch
  ### IMPORTANT: Also run "helm dependency update ." in this directory if you update any dependency version.
  version: 1.2.4
  repository: "https://opensearch-project.github.io/helm-charts/"
- name: opensearch-dashboards
  ### IMPORTANT: Also run "helm dependency update ." in this directory if you update any dependency version.
  version: 1.0.6
  repository: "https://opensearch-project.github.io/helm-charts/"

values.yml

opensearch: 
  config:
    opensearch.yml:
      plugins.security.ssl.transport.pemcert_filepath: certs/tls.crt
      plugins.security.ssl.transport.pemkey_filepath: certs/tls.key
      plugins.security.ssl.transport.pemtrustedcas_filepath: certs/tls.crt
      plugins.security.ssl.transport.enforce_hostname_verification: false
      plugins.security.ssl.http.enabled: true
      plugins.security.ssl.http.pemcert_filepath: certs/tls.crt
      plugins.security.ssl.http.pemkey_filepath: certs/tls.key
      plugins.security.ssl.http.pemtrustedcas_filepath: certs/tls.crt
      plugins.security.allow_unsafe_democertificates: false
      plugins.security.authcz.admin_dn:
        - CN=<domain>
      plugins.security.nodes_dn:
        - 'CN=<domain>'

  persistence:
    enabled: true
    # Set to false to disable the `fsgroup-volume` initContainer that will update permissions on the persistent disk.
    enableInitChown: true
    labels:
      # Add default labels for the volumeClaimTemplate of the StatefulSet
      enabled: false
    # OpenSearch Persistent Volume Storage Class
    # If defined, storageClassName: <storageClass>
    # If set to "-", storageClassName: "", which disables dynamic provisioning
    # If undefined (the default) or set to null, no storageClassName spec is
    #   set, choosing the default provisioner.  (gp2 on AWS, standard on
    #   GKE, AWS & OpenStack)
    #
    storageClass: csi-disk
    accessModes:
      - ReadWriteOnce
    size: 25Gi
    annotations:
      everest.io/disk-volume-type: SSD

  # Enabling this will publically expose your OpenSearch instance.
  # Only enable this if you have security enabled on your cluster
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx-external
      # kubernetes.io/ingress.class: nginx
      # kubernetes.io/tls-acme: "true"
    path: /
    hosts:
      - <domain>
    tls:
      - secretName: opensearch-tls

  extraVolumes:
    - name: certificate
      secret:
        secretName: opensearch-tls

  extraVolumeMounts:
    - name: certificate
      mountPath: "/usr/share/opensearch/config/certs"
      readOnly: true
  
  extraEnvs:
    - name: DISABLE_INSTALL_DEMO_CONFIG
      value: "true"


opensearch-dashboards:
  # Cluster-internal connection
  opensearchHosts: "https://opensearch-cluster-master:9200"
  imageTag: ""

  #config:
    # Default OpenSearch Dashboards configuration from docker image of Dashboards
    # opensearch_dashboards.yml:
      # opensearch.ssl.verificationMode: none
      # server.ssl.enabled: false

  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx-external
      #kubernetes.io/tls-acme: "true"
    hosts:
      - host: <dashboards-domain>
        paths:
          - path: /
            backend:
              serviceName: opensearch-dashboards-dev
              servicePort: 5601
    tls: 
      - secretName: opensearch-dashboard-cert

Both ingress rules deliver the right certificate.

If I connect over Dashboards to OpenSearch everything is ok (makes sense since verification disabled).
If I connect over my Browser (Firefox) I can see that the certificate is correct and valid but I get 502 Bad Gateway from Nginx.

in the ingress-nginx-external-controller for the ingress from OpenSearch I get the following errors:

2021/11/10 14:41:38 [error[] 4261#4261: *32847247 upstream prematurely closed connection while reading response header from upstream, client: 100.125.67.47, server: <domain>, request: "GET / HTTP/2.0", upstream: "http://172.16.0.61:9200/", host: "<domain>"
2021/11/10 14:41:38 [error[] 4261#4261: *32847247 upstream prematurely closed connection while reading response header from upstream, client: 100.125.67.47, server: <domain>, request: "GET / HTTP/2.0", upstream: "http://172.16.0.108:9200/", host: "<domain>"
2021/11/10 14:41:38 [error[] 4261#4261: *32847247 upstream prematurely closed connection while reading response header from upstream, client: 100.125.67.47, server: <domain>, request: "GET / HTTP/2.0", upstream: "http://172.16.0.139:9200/", host: "<domain>"

Here are the OpenSearch pods:

┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Pods(opensearch)[4] ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ NAME                                                                   PF       READY             RESTARTS STATUS               CPU       MEM↓       %CPU/R       %CPU/L       %MEM/R       %MEM/L IP                  NODE                  AGE           │
│ opensearch-cluster-master-1                                            ●        1/1                      0 Running               16       1246            1          n/a         1246          n/a 172.16.0.61         10.202.102.43         74m           │
│ opensearch-cluster-master-2                                            ●        1/1                      0 Running               28       1245            2          n/a         1245          n/a 172.16.0.108        10.202.102.36         75m           │
│ opensearch-cluster-master-0                                            ●        1/1                      0 Running               25       1131            2          n/a         1131          n/a 172.16.0.139        10.202.102.159        74m           │

Every OpenSearch-Pod has more or less the same error message:

opensearch [2021-11-10T14:51:26,174[][ERROR[][o.o.s.s.h.n.SecuritySSLNettyHttpServerTransport[] [opensearch-cluster-master-2[] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310... (shorted)
opensearch io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310... (shorted)
opensearch     at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1261) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at java.lang.Thread.run(Thread.java:832) [?:?]
opensearch [2021-11-10T14:51:26,175[][WARN [][o.o.h.AbstractHttpServerTransport[] [opensearch-cluster-master-2[] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.16.0.108:9200, remoteAddress=/172.16.0.52:46764}
opensearch io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310... (shorted)
opensearch     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:478) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at java.lang.Thread.run(Thread.java:832) [?:?]
opensearch Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310... (shorted)
opensearch     at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1261) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1329) ~[netty-handler-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:508) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[]
opensearch     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447) ~[netty-codec-4.1.59.Final.jar:4.1.59.Final[]
opensearch     ... 16 more

The decoded shorted message is hex format and is an HTTP-Request:

GET / HTTP/1.1
Host: <domain>
X-Request-ID: 9c50cec41af24277b5e869f99e657733
X-Real-IP: 100.125.67.47
X-Forwarded-For: 100.125.67.47
X-Forwarded-Host: <domain>
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Scheme: https
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:94.0) Gecko/20100101 Firefox/94.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
accept-language: de,en-US;q=0.7,en;q=0.3
accept-encoding: gzip, deflate, br
authorization: Basic YWRtaW46YWRtaW4=
upgrade-insecure-requests: 1
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: none
sec-fetch-user: ?1
cache-control: max-age=0

Has this something to do with timeouts or is still something misconfigured?

I think I forget to mention that the kubernetes cluster is only accessible through an VPN-Tunnel.
But the <domain> is accessible from everywhere.
Could this also be part of the problem?

Topic		Replies	Views
How to set up certificates for OpenSearch on Kubernetes? Security	7	2146	January 4, 2023
(CA) TLS Certificates in Kubernetes, how do you generate/maintain them Kubernetes	1	244	December 16, 2023
Encrypted communication between nodes and client Security	8	390	August 9, 2021
Admin Certificate Generation in K8s Security configure	2	519	April 7, 2023
Unable to configure newly generated SSL certificates on a 3 node opensearch cluster Security configure	3	514	April 20, 2022

K8s: Define SSL certificates via secrets

Related Topics