Getting "unable to verify the first certificate" error from OpenSearch Dashboards in k8s

Security
1

Hey everyone! I seem to be having some issues generating the certificates for OpenSearch. What I did was that I followed the “Generate certificates” guide and was able to get OpenSearch working just fine. I generated the certs using the script shown on the guide.

But the problem comes up when setting up TLS for OpenSearch Dashboards. I followed the Configure TLS for OpenSearch Dashboards. But when I deploy it to my k8s cluster OpenSearch Dashboards shows the following error:

{"type":"log","@timestamp":"2021-12-10T16:29:56Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: unable to verify the first certificate"}

After some searching around it seems that the error means that the issuer CA is missing on the OpenSearch Dashboards (opensearch-project/documentation-website/issues/144).

With all that said, the OpenSearch Dashboard TLS guide does indeed show the issuer CA on the demo config:

opensearch.ssl.certificateAuthorities: [ "/usr/share/opensearch-dashboards/config/root-ca.pem", "/usr/share/opensearch-dashboards/config/intermediate-ca.pem" ]

So coming from the OpenSearch TLS guide and generating all the certs using the script shown there, the generation of the intermediate-ca.pem cert seems to be missing from that script. How is this intermediate-ca.pem generated?

2

@derek-diaz The intermediate CA cert is used to sign certificates on behalf of Root CA. This is common for commercially signed certificates. It is not mandatory to have an intermediate CA certificate. In the local network, you can use Root CA as a signing authority.
That’s why the mentioned procedure doesn’t contain the generation of an intermediate CA certificate.

The reported error could mean that OpenSearch Dashboards possibly can’t access the Root CA or Root CA is not the one that signed the TLS certificate.

Could you share your OpenSearch Dashboards and OpenSearch yaml config files?

3

Hey @pablo here’s my yaml file. I currently mount all the certs using secretMounts. This is part of the values.yaml for Helm. When I check the files in the pod, they are mounted in the correct location. So maybe I am missing something else?

opensearch-dashboards:
  imageTag: "1.2.0"
  extraEnvs:
    - name: "NODE_OPTIONS"
      value: "--max-old-space-size=1800"
  opensearchHosts: "https://opensearch-cluster-master:9200"

  opensearchAccount:
    secret: "opensearch-dashboard-credentials"

  secretMounts:
    - name: opensearch-certificate-root
      secretName: opensearch-certificate-root
      path: /usr/share/opensearch-dashboards/config/certs/root
    - name: opensearch-certificate-client
      secretName: opensearch-certificate-client
      path: /usr/share/opensearch-dashboards/config/certs/client
    - name: opensearch-certificate-client-key
      secretName: opensearch-certificate-client-key
      path: /usr/share/opensearch-dashboards/config/certs/client-key
      
  config:
    opensearch_dashboards.yml: |
      opensearch.ssl.verificationMode: full
      opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]
      opensearch_security.cookie.secure: true
      opensearch.username: "some username goes here"
      opensearch.password: "some password goes here"
      server.name: "OpenSearch"
      server.ssl.enabled: true
      server.ssl.certificate: /usr/share/opensearch-dashboards/config/certs/client/client.pem
      server.ssl.key: /usr/share/opensearch-dashboards/config/certs/client-key/client-key.pem
      opensearch.ssl.certificateAuthorities: [ "/usr/share/opensearch-dashboards/config/certs/root/root-ca.pem" ]
4

Quick update, it seems that the helm chart for OpenSearch Dashboard is appending an additional | on the configs:
image

5

The OpenSearch chart config has a similar config to OpenSearch Dashboards and it is not adding that extra |:

image

I just changed what I had in the previous post to this:

config:
    opensearch_dashboards.yml:
      opensearch:
        requestHeadersWhitelist: ["securitytenant", "Authorization"]
        username: "some username"
        password: "some password"
        ssl:
          certificateAuthorities: /usr/share/opensearch-dashboards/config/certs/root/root-ca.pem
          verificationMode: full
      opensearch_security:
        cookie.secure: false
      server:
        name: "awesome server name"
        ssl:
          enabled: true
          certificate: /usr/share/opensearch-dashboards/config/certs/client/client.pem
          key: /usr/share/opensearch-dashboards/config/certs/client-key/client-key.pem

And it immediately picked up the config changes. OpenSearch Dashboards now boots correctly with SSL.

Is this a bug?

6

@derek-diaz the value.yaml for opensearch-dashboards has this section commented out by default.
Where did you download that helm chart?

7

That values file is a file I created to override the default values. This is to enable SSL on both OpenSearch and OpenSearch Dashboards on Kubernetes. The only chart that seems to be having this issue is the OpenSearch Dashboards Chart.