Issue with OpenSearch dashboard with ssl certificate

Security
1

Hi @all,
we are trying to configure Opensearch-Dashboard with self signed SSL certificate for opensearch.

Here we are using letsencrypt certificate.
Server details:

Server: Rocky Linux, version: 8.5
OpenSearch config location: /oss/bin/opensearch
OpenSearch dashboard config location: /oss/bin/opensearch-dashboard/config/

SSL details:

files location: /etc/letsencrypt/archive/reportplus.engrid.in/

This files are symlink with opensearch config location.

lrwxrwxrwx. 1 oss oss    55 Mar 24 17:35 cert1.pem -> /etc/letsencrypt/archive/reportplus.engrid.in/cert1.pem
lrwxrwxrwx. 1 oss oss    56 Mar 24 17:35 chain1.pem -> /etc/letsencrypt/archive/reportplus.engrid.in/chain1.pem
lrwxrwxrwx. 1 oss oss    60 Mar 24 17:35 fullchain1.pem -> /etc/letsencrypt/archive/reportplus.engrid.in/fullchain1.pem
lrwxrwxrwx. 1 oss oss    58 Mar 24 17:35 privkey1.pem -> /etc/letsencrypt/archive/reportplus.engrid.in/privkey1.pem

Opensearch is working fine with these SSL related files.

Curl status for opensearch:

$ curl -XGET https://reportplus.engrid.in:9200/ -u 'admin:admin' --cacert /oss/bin/opensearch/config/cert1.pem
{
  "name" : "node-1",
  "cluster_name" : "Subhasree-Reporting",
  "cluster_uuid" : "amGlvdSASMKF3lfHT4jrdQ",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.2.4",
    "build_type" : "tar",
    "build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
    "build_date" : "2022-01-14T03:38:06.881862Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

We are trying to configure the OpenSearch-dashboard with this SSL certificate files.

Here is the opensearch-dashboard configuration.

# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0


# Description:
# Default configuration for OpenSearch Dashboards

server.host: "0.0.0.0"
opensearch.hosts: ["https://localhost:9200"]
opensearch.ssl.verificationMode: certificate
opensearch.username: "admin"
opensearch.password: "admin"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]

server.ssl.enabled: true
server.ssl.certificate: /oss/bin/opensearch/config/cert1.pem
server.ssl.key: /oss/bin/opensearch/config/privkey1.pem
opensearch.ssl.certificateAuthorities: /oss/bin/opensearch/config/cert1.pem
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
opensearch_security.cookie.secure: true

After execute the /bin/opensearch-dashboards, this error is generated.

  log   [09:44:18.471] [info][plugins-service] Plugin "visTypeXy" is disabled.
  log   [09:44:18.680] [info][plugins-system] Setting up [44] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,securityDashboards,reportsDashboards,indexManagementDashboards,anomalyDetectionDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,ganttChartDashboards,visualize,queryWorkbenchDashboards,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,observabilityDashboards,discover,savedObjectsManagement,bfetch]
  log   [09:44:19.162] [info][savedobjects-service] Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations...
  log   [09:44:19.224] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate
  log   [09:44:19.240] [error][savedobjects-service] Unable to retrieve version information from OpenSearch nodes.
  log   [09:44:21.725] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate
  log   [09:44:24.240] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate
  log   [09:44:26.751] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate

Change the configuration in this state,

# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0


# Description:
# Default configuration for OpenSearch Dashboards

server.host: "0.0.0.0"
opensearch.hosts: ["https://localhost:9200"]
opensearch.ssl.verificationMode: certificate
opensearch.username: "admin"
opensearch.password: "admin"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]

server.ssl.enabled: false
#server.ssl.certificate: /oss/bin/opensearch/config/cert1.pem
#server.ssl.key: /oss/bin/opensearch/config/privkey1.pem
opensearch.ssl.certificateAuthorities: /oss/bin/opensearch/config/cert1.pem
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
# Use this setting if you are running kibana without https
opensearch_security.cookie.secure: false

On starting the OpenSearch-Dashboard, getting this error.

 log   [10:03:19.517] [info][plugins-service] Plugin "visTypeXy" is disabled.
  log   [10:03:19.714] [info][plugins-system] Setting up [44] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,securityDashboards,reportsDashboards,indexManagementDashboards,anomalyDetectionDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,ganttChartDashboards,visualize,queryWorkbenchDashboards,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,observabilityDashboards,discover,savedObjectsManagement,bfetch]
  log   [10:03:20.203] [info][savedobjects-service] Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations...
  log   [10:03:20.266] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate
  log   [10:03:20.278] [error][savedobjects-service] Unable to retrieve version information from OpenSearch nodes.
  log   [10:03:22.768] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate
  log   [10:03:25.268] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate

If anyone could point me in the right direction that would be great help.
Thanks

2

@ChandanHaldar As per documentation opensearch.ssl.certificateAuthorities expects an array.

Try the following.

opensearch.ssl.certificateAuthorities: ["/oss/bin/opensearch/config/cert1.pem"]

Is the cert1.pem Root CA or OpenSearch node certificate? Any intermediate CA certificates must be also included.

3

@pablo thank you for this guidense.
I have change the configuration file’s entry.
The new configuration is:

server.port: 5601
server.host: "0.0.0.0"
opensearch.hosts: ["https://reportplus.engrid.in:9200"]
opensearch.ssl.verificationMode: full
opensearch.username: "admin"
opensearch.password: "admin"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]

server.ssl.enabled: true
server.ssl.certificate: /oss/bin/opensearch/config/cert1.pem
server.ssl.key: /oss/bin/opensearch/config/privkey1.pem
opensearch.ssl.certificateAuthorities: ["/oss/bin/opensearch/config/cert1.pem"]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
# Use this setting if you are running kibana without https
opensearch_security.cookie.secure: true

On starting opensearch-dashboard, getting this error:

  log   [06:17:52.784] [info][plugins-service] Plugin "visTypeXy" is disabled.
  log   [06:17:52.992] [info][plugins-system] Setting up [44] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,securityDashboards,reportsDashboards,indexManagementDashboards,anomalyDetectionDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,ganttChartDashboards,visualize,queryWorkbenchDashboards,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,observabilityDashboards,discover,savedObjectsManagement,bfetch]
  log   [06:17:53.479] [info][savedobjects-service] Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations...
  log   [06:17:53.554] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate
  log   [06:17:53.568] [error][savedobjects-service] Unable to retrieve version information from OpenSearch nodes.
  log   [06:17:56.064] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate
  log   [06:17:58.551] [error][data][opensearch] [ConnectionError]: unable to verify the first certificate

If anyone could point me in the right direction that would be help.
Thanks…

4

Hi @all,
I have make changes in opensearch dashboard configuration file.
Configuration file;

# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0


# Description:
# Default configuration for OpenSearch Dashboards
server.port: 5601
server.host: "0.0.0.0"
opensearch.hosts: ["https://reportplus.engrid.in:9200"]
opensearch.ssl.verificationMode: none
opensearch.username: "admin"
opensearch.password: "admin"
opensearch.requestHeadersWhitelist: [ authorization,securitytenant ]

server.ssl.enabled: true
server.ssl.certificate: /oss/bin/opensearch/config/cert1.pem
server.ssl.key: /oss/bin/opensearch/config/privkey1.pem
opensearch.ssl.certificateAuthorities: ["/oss/bin/opensearch/config/fullchain1.pem"]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.enable_global: true
opensearch_security.multitenancy.tenants.enable_private: true
opensearch_security.multitenancy.tenants.preferred: ["Private", "Global"]
opensearch_security.multitenancy.enable_filter: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
# Use this setting if you are running kibana without https
opensearch_security.cookie.secure: true

Now OpenSearch dashboard is open with SSL.
Can anyone please help me for how to use full instead of none mode for “opensearch.ssl.verificationMode” Here.
if anyone could point me in the right direction that would be help.

5

@ChandanHaldar OpenSearch Dashboards is trying to verify OpenSearch certificate against the node name (reportplus.engrid.in) and certificate chain.

In short reportplus.engrid.in has to be present in CN and SAN of the OS certificate and signed by RootCA from fullchain1.pem

6

@ChandanHaldar have you figured out the problem? I ran into the same issue

7

HI …
you manage to find any solution , I face the same issue now

8

@vinot Could you open a new thread and share your configs and OpenSearch version?

9

Hi, Is this working ?

10

@ChandanHaldar If you need a help with your OpenSearch Dashboards, please open a new thread and share your config files.