[ConnectionError]: unable to verify the first certificate

Security
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.8.0

Describe the issue:
When attempting to use HTTPS for opensearch dashboard the following error is present in the logs:
[ConnectionError]: unable to verify the first certificate

This is resulting in the dashboard webpage displaying an error stating that the dashboard is not ready (however the webpage shows that the connection is secure).

The certificate I am using is a wild card that matches my nodes and my dashboards domain. It also contains both Client and Server Auth under the EKU.

SSL for my nodes is working correctly using the same certificates.

If I drop opensearch.ssl.verificationMode to None then SSL works, but dropping it to Certificate still fails.
Configuration:
opensearch.hosts: [“my_node1.net”]
opensearch.ssl.verificationMode: full
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
server.ssl.enabled: true
server.ssl.certificate: /usr/share/opensearch-dashboards/certs/cert.pem
server.ssl.key: /usr/share/opensearch-dashboards/certs/cert-key.pem
opensearch.ssl.certificateAuthorities: [“/usr/share/opensearch-dashboards/certs/root-ca.pem”, “/usr/share/opensearch-dashboards/certs/intermediate-ca.pem”]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: true
server.host: ‘my_dashbooard.net’

plugins.security.ssl.transport.pemcert_filepath: esnode.pem
plugins.security.ssl.transport.pemkey_filepath: esnode-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: esnode.pem
plugins.security.ssl.http.pemkey_filepath: esnode-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: root-ca.pem
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.nodes_dn:

  • ‘myDNs’

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [“.plugins-ml-model-group”, “.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.ql-datasources”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”, “.opensearch-knn-models”]
node.max_local_storage_nodes: 3

Relevant Logs or Screenshots:
:[“error”,“opensearch”,“data”],“pid”:120439,“message”:“[ConnectionError]: unable to verify the first certificate”}

2

Hi @baazzaar

Could you share the output of the command below?

openssl x509 -in /usr/share/opensearch-dashboards/certs/cert.pem -noout -purpose

4

Please try to concatenate both Intermediate and root-ca in one file:
opensearch.ssl.certificateAuthorities: [“/usr/share/opensearch-dashboards/certs/root-ca.pem”, “/usr/share/opensearch-dashboards/certs/intermediate-ca.pem”]

Or try using just the root-ca in that option. The intermediate should be included in the server certificate defined in esnode.pem

5

Hi @Eugene7
Thanks for taking a look!

I have tried concatenating the intermediate into both the root CA file and Server file, but in both cases the issue remains.

Here is the output of the openssl command you requested:

Certificate purposes:
SSL client : Yes
SSL client CA : No
SSL server : Yes
SSL server CA : No
Netscape SSL server : Yes
Netscape SSL server CA : No
S/MIME signing : No
S/MIME signing CA : No
S/MIME encryption : No
S/MIME encryption CA : No
CRL signing : No
CRL signing CA : No
Any Purpose : Yes
Any Purpose CA : Yes
OCSP helper : Yes
OCSP helper CA : No
Time Stamp signing : No
Time Stamp signing CA : No
6

Hi @baazzaar

As per documentation, server.ssl.certificate specifies the full path to a valid client certificate for your OpenSearch cluster. Please try to make a client-only certificate.

7

Hi @Eugene7,

Interesting. I might be misunderstanding but would the Opensearch Dashboard not be the server in this scenario? With the browser being the client?

Just to make sure I am understanding this correctly, the certificates used in the dashboard serve two purposes, firstly to allow an SSL connection for HTTPS when end-users are accessing the dashboard UI. Secondly, for authenticating the Opensearch nodes with the dashboard. For this to work the DN in the dashboard certificate needs to match the hostnames of the nodes.
Please correct me if I am incorrect.

8

Hi @baazzaar

Did you use RootCA and intermediate certificates to sign the OpenSearch node certificate or OpenSearch Dashboard certificate?

Can you please verify the certification path of the OS node certificate and confirm that Intermediate and RootCA defined in the OS Dashboards are the same? You should verify only the certificate defined in plugins.security.ssl.http.pemcert_filepath

9

Hi @Eugene7

I am using the same Certificates for the dashboard and the nodes as they are wildcard server/client certificates.

10

Could you share the output of the following commands? Please delete or change any sensitive data before sharing.

openssl x509 -in root-ca.pem -text -noout
openssl x509 -in intermediate-ca.pem -text -noout
openssl x509 -in cert.pem -text -noout
11

Hi @Eugene7

I have PM’ed you the requested information.

12

Hi @baazzaar

  1. Did you concatenate the intermediate certificate to the node certificate in plugins.security.ssl.http.pemcert_filepath in opensearch.yml (not the transport)?

  2. How many OpenSearch nodes did you define in opensearch.hosts in opensearch_dashboards.yml? Is my_node1.net the only one?

  3. Did you remove the value “/usr/share/opensearch-dashboards/certs/intermediate-ca.pem” from opensearch.ssl.certificateAuthorities after concatenation ?

  4. Did you restart OpenSearch and OpenSearch Dashboards after concatenating intermediate certificates and changing the configuration?

13

I have finally managed to get this working. I will detail my exact config here in the hopes that it may be of use to someone else:

opensearch.yml

cluster.name: foo

network.host: {{ IP OF EACH NODE }}

plugins.security.ssl.transport.pemcert_filepath: {{ Relative file path. By default Opensearch will look under /usr/share/opensearch/config/ }}
plugins.security.ssl.transport.pemkey_filepath: {{ Relative file path. By default Opensearch will look under /usr/share/opensearch/config/ }}
plugins.security.ssl.transport.pemtrustedcas_filepath: {{ Relative file path. By default Opensearch will look under /usr/share/opensearch/config/ }}
plugins.security.ssl.transport.enforce_hostname_verification: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: {{ Relative file path. By default Opensearch will look under /usr/share/opensearch/config/ }}
plugins.security.ssl.http.pemkey_filepath: {{ Relative file path. By default Opensearch will look under /usr/share/opensearch/config/ }}
plugins.security.ssl.http.pemtrustedcas_filepath: {{ Relative file path. By default Opensearch will look under /usr/share/opensearch/config/ }}
plugins.security.allow_unsafe_democertificates: false
plugins.security.allow_default_init_securityindex: true
plugins.security.nodes_dn:
  - '{{ MY DN }}'

plugins.security.audit.type: internal_opensearch
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.restapi.roles_enabled: ["all_access", "security_rest_api_access"]
plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".plugins-ml-model-group", ".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".ql-datasources", ".opendistro-asynchronous-search-response*", ".replication-metadata-store", ".opensearch-knn-models"]
node.max_local_storage_nodes: 3

pemtrustedcas_filepath Contains both the Root CA and the intermediate.

opensearch_dashboards.yml

---
opensearch.hosts: [{{ List of Nodes }}]
opensearch.ssl.verificationMode: full
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
server.ssl.enabled: true
server.ssl.certificate: {{ Full file path. For me this was /usr/share/opensearch-dashboards/config/ }}
server.ssl.key: {{ Full file path. For me this was /usr/share/opensearch-dashboards/config/ }}
opensearch.ssl.certificateAuthorities: ["{{ Full file path. For me this was /usr/share/opensearch-dashboards/config/ }}"]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: true
server.host: '{{ IP of Dashboard }}'

certificateAuthorities Contains both the Root CA and the intermediate.

Thanks for the help @Eugene7