Bring your own certificate using Helm (OpenSearch) with separate pods

OpenDistro
1

I am having problems logistically trying to understand the order of operation for locking down OpenSearch with my own supplied certificates (from a 3rd party).

This is where I need some conceptual help.

I have 4 pods running, master, data, client, dashboard.
I want to secure the dashboard with my own certificate.

Everything is deployed via Helm (branch: 1.0.0, https://github.com/opensearch-project/helm-charts.git)

I created a Kubernetes secret with my public certificate, intermediate chain and private key:
kubectl create secret generic iots-cert --from-file=sandbox.my.domain.chain --from-file=sandbox.my.domain.pub --from-file=sandbox.my.domain.key

This is where conceptually I need help.

If I enable TLS on the dashboard, do I need to do anything with the master, data and client pods?

  • Technically, are they not only reachable from within the cluster?
    • I don’t think I need to do anything outside of the default
  • Do each of the master, data and client need ingresses?
    • I don’t think so, as they are all on the same cluster

This is what I have done to enable TLS, but I get an error message when the dashboard starts up.

secretMounts: 
  - name: iots-cert
    secretName: iots-cert
    path: /usr/share/dashboards/certs

config: 
  ## Default OpenSearch Dashboards configuration from docker image of Dashboards
  opensearch_dashboards.yml:
    server:
      name: dashboards
      host: 0.0.0.0

      ## Dashboards TLS Config (Ensure the cert files are present before enabling SSL
      ssl:
        enabled: true
        key: /usr/share/dashboards/certs/sandbox.my.domain.key
        certificate: /usr/share/dashboards/certs/sandbox.my.domain.pub

    # determines how dashboards will verify certificates (needs to be none for default opensearch certificates to work)
    opensearch:
      ssl:
        certificateAuthorities: /usr/share/dashboards/certs/sandbox.my.domain.chain

ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
  hosts:
    - host: kibana.ingress.sandboxbsh-a.iotdevk8s.shoot.canary.k8s.domain.com
      paths:
        - path: /
          backend:
            serviceName: opensearch-cluster-client
            servicePort: 9200
            # servicePort: 80
  tls:
    - secretName: iots-cert
      hosts:
        - logs-test.sandbox.my.domain

Unfortunately, this yields this error:

{"type":"log","@timestamp":"2021-11-29T22:00:28Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: unable to verify the first certificate"}

If I comment out the config section, I can ssh into the box and verify the 3 sandbox files are present and accounted for.
I have also verified the contents of these files are correct.
For troubleshooting, I reversed the pub and the chain, I get a clear error starting, so I know I have those correct.

So I am not sure if this error is referring to the certs I have specified, or when it tries to use those certs to possibly access other components in the setup.
Give the error message indicates data, I wasn’t sure if it is an error trying to reach a data node.
Though, I thought Kibana (Dashboard) only accesses the master and never a data node directly.

Any suggestions?

Thank you,
David