Issue with OpenSearch dashboard TLS Configuration

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch - 2.3.0
OS - Linux (ubuntu 18.04)

Hi @all,
we are trying to configure Opensearch-Dashboard with self signed SSL certificate for opensearch.

Describe the issue:

OpenSearch config location: /home/aiml/opensearch-2.3.0/bin
OpenSearch dashboard config location: /home/aiml/opensearch-dashboards-2.3.0/bin

Opensearch is working fine with these SSL related files.

Curl status for opensearch:

{
“name” : “CNAS-AIML”,
“cluster_name” : “opensearch”,
“cluster_uuid” : “gWx3JHPeQOWZW1llka_6tA”,
“version” : {
“distribution” : “opensearch”,
“number” : “2.3.0”,
“build_type” : “tar”,
“build_hash” : “6f6e84ebc54af31a976f53af36a5c69d474a5140”,
“build_date” : “2022-09-09T00:07:12.137133581Z”,
“build_snapshot” : false,
“lucene_version” : “9.3.0”,
“minimum_wire_compatibility_version” : “7.10.0”,
“minimum_index_compatibility_version” : “7.0.0”
},
}

We are trying to configure the OpenSearch-dashboard with this SSL certificate files.

Configuration:
Here is the opensearch-dashboard configuration.

server.port: 5601
server.host: “0.0.0.0”
opensearch.hosts: [https://localhost:9200]
opensearch.username: “admin”
opensearch.password: “Accenture@123”
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch.ssl.verificationMode: full
server.ssl.enabled: true
server.ssl.certificate: /home/aiml/opensearch-2.3.0/config/node1.pem
server.ssl.key: /home/aiml/opensearch-2.3.0/config/node1-key.pem
opensearch.ssl.certificateAuthorities: [ “/home/aiml/opensearch-2.3.0/config/root-ca.pem” ]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: true

After execute the /bin/opensearch-dashboards, this error is generated.

aiml@CNAS-AIML:~/opensearch-dashboards-2.3.0/bin$ ./opensearch-dashboards

  • log [09:21:20.879] [info][plugins-service] Plugin “visTypeXy” is disabled.*
  • log [09:21:20.887] [info][plugins-service] Plugin “wizard” is disabled.*
  • log [09:21:20.936] [warning][config][deprecation] “opensearch.requestHeadersWhitelist” is deprecated and has been replaced by “opensearch.requestHeadersAllowlist”*
  • log [09:21:21.067] [info][plugins-system] Setting up [46] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,securityDashboards,indexManagementDashboards,anomalyDetectionDashboards,queryWorkbenchDashboards,notificationsDashboards,reportsDashboards,charts,legacyExport,embeddable,dashboard,visualizations,visTypeTimeseries,visTypeVislib,visTypeVega,visTypeTimeline,timeline,visTypeMarkdown,visTypeTagcloud,visTypeTable,visTypeMetric,tileMap,regionMap,customImportMapDashboards,inputControlVis,observabilityDashboards,ganttChartDashboards,visualize,discover,savedObjectsManagement,bfetch]*
  • log [09:21:21.626] [info][savedobjects-service] Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations…*
  • log [09:21:21.663] [error][data][opensearch] [ConnectionError]: self signed certificate*
  • log [09:21:21.670] [error][savedobjects-service] Unable to retrieve version information from OpenSearch nodes.*
  • log [09:21:24.161] [error][data][opensearch] [ConnectionError]: self signed certificate*
  • log [09:21:26.662] [error][data][opensearch] [ConnectionError]: self signed certificate*
  • log [09:21:29.164] [error][data][opensearch] [ConnectionError]: self signed certificate*

When we change the configuration to this state:

server.port: 5601
server.host: “0.0.0.0”
opensearch.hosts: [https://localhost:9200]
opensearch.username: “admin”
opensearch.password: “Accenture@123”
opensearch.requestHeadersWhitelist: [authorization, securitytenant]
opensearch.ssl.verificationMode: none
server.ssl.enabled: true
server.ssl.certificate: /home/aiml/opensearch-2.3.0/config/node1.pem
server.ssl.key: /home/aiml/opensearch-2.3.0/config/node1-key.pem
opensearch.ssl.certificateAuthorities: [ “/home/aiml/opensearch-2.3.0/config/root-ca.pem” ]
opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
opensearch_security.cookie.secure: true

On starting the OpenSearch-Dashboard, getting this error.

Web UI is working

Now OpenSearch dashboard is open with SSL.

Can anyone please help me for how to use full instead of none mode for “opensearch.ssl.verificationMode” Here.

If anyone could point me in the right direction that would be great help.

@pablo
Thanks

@Bindu Do you get the same message when you set certificate instead of full?

Hi @pablo if I set the opensearch.ssl.verificationMode: certificate/full, I am seeing the below error:

@Bindu How did you generate certificates for OpenSearch nodes?
Could you share the output of the below command?

openssl x509 -in /home/aiml/opensearch-2.3.0/config/root-ca.pem -text -noout

openssl x509 -in /home/aiml/opensearch-2.3.0/config/<OS_node_cert>.pem -text -noout

Hi @Pablo,

Below is the output of the root-ca.pem

openssl x509 -in /home/aiml/opensearch-2.3.0/config/root-ca.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
53:ac:51:18:a7:a1:f4:4a:2e:07:c7:20:77:9b:61:49:1f:90:98:67
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = XX, ST = YY, L = ZZ, O = AA, OU = IT, CN = xx.yy.cloudapp.azure
Validity
Not Before: Mar 7 05:46:49 2023 GMT
Not After : Mar 6 05:46:49 2025 GMT
Subject: C = XX, ST = YY, L = ZZ, O = AA, OU = IT, CN = xx.yy.cloudapp.azure.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:97:49:1c:5c:02:55:33:f8:52:65:50:cc:87:fc:
b0:19:7e:a2:cb:5d:6f:93:6a:9c:1d:87:a7:4c:a6:
82:c9:47:98:a0:3a:87:eb:fc:1d:2f:e7:73:12:54:
b2:8b:17:89:94:92:75:2d:4f:72:d4:3f:1d:12:b0:
c1:d3:87:f9:ad:e7:d4:3a:e4:05:15:b1:2e:36:db:
a7:51:e1:8a:84:94:51:74:b1:df:b0:5d:2c:3c:a6:
1e:2d:64:89:5d:2d:c9:1c:db:8b:11:cd:66:71:0e:
7e:2c:c5:05:71:7d:ed:fe:0c:03:65:b7:6c:fe:65:
bd:74:fa:2f:b0:a2:91:73:2c:fb:f6:c9:c9:53:9e:
72:f0:bb:87:cd:7b:57:04:66:fd:26:a1:2d:a9:bb:
a8:af:db:9c:70:fd:58:68:fc:11:4f:60:ee:9a:e6:
1a:62:7a:40:a8:22:55:ee:28:ce:6a:51:6d:cb:bd:
20:de:55:38:34:a2:7e:e5:53:1c:73:f4:09:eb:b4:
b8:56:08:42:42:76:3c:bc:2a:12:b6:f9:a2:db:31:
b3:04:03:e5:bd:fd:06:a2:19:49:a4:3f:52:03:56:
04:8d:10:73:05:de:37:bc:cd:1b:ed:36:5a:50:3d:
65:de:a0:32:b4:e4:82:3c:18:5e:4e:4e:46:25:71:
d1:63
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
02:94:E7:00:A4:D0:47:E6:A4:CC:BD:05:A4:FE:DF:CB:DE:33:74:41
X509v3 Authority Key Identifier:
keyid:02:94:E7:00:A4:D0:47:E6:A4:DA:BD:05:A4:FE:DF:CB:DE:AA:74:41

        X509v3 Basic Constraints: critical
            CA:TRUE
Signature Algorithm: sha256WithRSAEncryption
     04:54:16:2a:64:0a:7d:a1:a1:1a:7f:6d:9f:a5:9e:ee:91:95:
     db:41:22:c9:ad:ee:0e:54:36:47:2f:e2:6f:d9:28:30:13:9c:
     d0:e4:8c:54:e0:03:a2:56:34:09:2e:3a:16:fe:2d:11:58:7c:
     61:24:d4:a2:ec:33:a4:f7:2b:9d:ed:9f:d3:18:3b:04:2b:f4:
     96:b6:3d:cd:86:e1:4b:69:a8:7d:3f:4b:50:da:b8:30:09:6c:
     ef:7d:15:88:e1:3d:46:0b:ab:58:3b:ce:da:1f:15:b0:c8:6c:
     0e:c9:e4:f3:74:ad:35:1d:f1:f6:46:ea:1e:b5:b0:b7:d2:5b:
     19:3a:56:21:08:8f:22:75:2e:e0:91:c2:b7:ef:3f:2b:1f:85:
     80:b5:70:fb:29:c3:69:e3:1b:27:3a:35:07:3f:32:66:e7:3f:
     c0:5f:4f:b5:b8:d5:d2:6c:0d:b3:65:cf:8a:2d:cd:ca:ea:4d:
     5d:06:4f:ce:e8:6f:3d:fc:84:36:82:83:8e:1a:4a:85:62:ec:
     0f:c6:f7:03:9e:1a:f9:59:b2:d3:94:1f:79:eb:a8:b7:1e:92:
     0a:c2:39:fb:62:62:30:24:94:5e:a1:af:f1:72:54:15:37:b3:
     df:69:3c:d7:7c:86:b1:5c:72:9b:aa:a2:3b:3f:b6:be:f3:b9:
     40:dc:e6:d2

openssl x509 -in /home/aiml/opensearch-2.3.0/config/node1.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
1f:8e:da:a4:a7:f9:65:1d:42:3b:fe:fe:fa:3e:7e:2c:35:10:93:8d
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = XX, ST = YY, L = ZZ, O = AA, OU = IT, CN = xx.yy.cloudapp
Validity
Not Before: Mar 7 05:54:04 2023 GMT
Not After : Mar 6 05:54:04 2025 GMT
Subject: C = XX, ST = YY, L = ZZ, O = AA, OU = IT, CN = xx.yy.cloudapp.azure.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:c3:dd:ed:51:fe:7b:97:91:82:a0:6a:60:05:28:
f1:b6:01:78:4b:47:31:b4:d9:6e:91:2d:94:b0:f4:
1b:60:67:19:16:8d:22:18:5d:8f:10:ee:e2:f2:e3:
a1:2f:78:1c:70:4d:1e:15:00:1c:d6:1e:f8:10:8d:
5a:48:5e:0a:92:e2:75:02:47:b2:01:a9:16:58:62:
fd:07:4f:b8:50:05:85:8a:2b:01:64:de:60:cf:38:
db:de:05:91:10:e7:ba:72:7f:c4:c4:6d:81:1e:02:
53:f2:ad:1c:1e:cb:54:91:7f:d4:60:7b:3e:18:5d:
08:d4:d5:2c:2a:82:d5:2e:f9:91:d1:c3:c8:1d:49:
29:69:c5:28:2a:06:f8:60:0c:9b:33:70:9e:8a:35:
9c:85:41:df:53:16:c0:23:9b:1e:69:2f:2a:68:46:
24:b3:a6:ee:da:8f:ab:ce:7e:be:17:2b:4d:c7:8f:
50:e2:f7:3f:4d:a2:53:ce:b0:b1:2b:97:08:55:47:
b5:75:a8:6e:e1:9a:15:73:4b:70:19:f9:ec:dd:45:
90:98:25:22:40:cc:0c:a6:6c:e0:1b:83:e0:c4:b3:
40:33:3c:84:88:5c:cb:cd:88:70:ec:a9:fa:bf:6f:
f7:2f:88:58:6f:41:d1:5f:ed:4b:63:32:1b:64:e6:
60:3b
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Alternative Name:
DNS:xx.yy.cloudapp.azure.com
Signature Algorithm: sha256WithRSAEncryption
7e:65:5c:f2:af:2b:1d:db:90:3b:ea:e7:0d:95:0d:fc:19:00:
b0:23:a5:16:7a:be:82:60:11:2d:58:56:03:d4:c2:75:ca:b6:
22:52:c4:2c:bc:56:13:9e:2f:f6:bc:c3:7d:1a:12:74:8f:9d:
bf:11:e4:ef:2d:76:de:d7:0b:9e:5c:8f:0b:7f:fe:07:14:01:
1f:7f:4b:60:3d:d7:25:3b:1e:21:68:a3:09:4b:8d:d3:18:72:
db:d7:bb:a6:80:2d:65:a4:38:f1:a2:87:12:7e:fd:1b:89:10:
b3:38:39:d4:bf:08:72:8e:9e:bd:02:62:88:fb:aa:e6:e3:cc:
e4:a4:6b:7b:eb:9f:66:9d:fe:db:2c:ba:65:88:d0:53:ab:18:
0e:02:5e:af:7d:01:e3:63:f6:11:5c:85:75:7e:c1:e6:b5:c0:
74:68:42:4c:97:bd:24:25:4f:73:3e:7e:00:7f:e3:25:75:42:
a4:69:c1:95:7c:c2:be:4a:ea:95:44:f4:b4:03:1a:62:d5:82:
64:9b:f0:c7:07:2c:5b:27:08:19:2a:41:91:dd:68:85:cb:e4:
08:9b:31:49:03:28:b1:e8:2c:8e:27:49:60:97:a1:24:39:c9:
64:10:ba:31:a5:8d:3f:1d:e1:cd:09:4c:7b:49:83:40:bf:1a:
2a:01:1d:ac

@Bindu How did you generate those certificates?

Hi @Pablo, these are the commands I used to generate certificates. I followed the doc: Generating self-signed certificates - OpenSearch documentation

openssl genrsa -out root-ca-key.pem 2048
openssl req -new -x509 -sha256 -key root-ca-key.pem -out root-ca.pem -days 730 openssl genrsa -out admin-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in admin-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out admin-key.pem
openssl req -new -key admin-key.pem -out admin.csr
openssl x509 -req -in admin.csr -CA root-ca-1.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out admin.pem -days 730
openssl genrsa -out node1-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in node1-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node1-key.pem
openssl req -new -key node1-key.pem -out node1.csr
echo ‘subjectAltName=DNS:CNAS-AIML’ > node1.ext
openssl x509 -req -in node1.csr -CA root-ca.pem -CAkey root-ca-key.pem -CAcreateserial -sha256 -out node1.pem -days 730 -extfile node1.ext
openssl x509 -subject -nameopt RFC2253 -noout -in node.pem

@Bindu Could you also share the opensearch.yml file? How many nodes do you have in the cluster?

I assume this is a mistake and your config files are in the config folder for both OS and OSD, is that correct?

Hi @Pablo, Yes I have pasted the wrong file location by mistake. The configs are in the Config folder for both OS and OSD.

Hi @Pablo,

Here is the Opensearch File Config, I have used only one node.

#----------------------------------- Memory -----------------------------------

#Lock the memory on startup:

#bootstrap.memory_lock: true

#Make sure that the heap size is set to about half the memory available
#on the system and that the owner of the process is allowed to use this
#limit.

#OpenSearch performs poorly when the system is swapping the memory.

#---------------------------------- Network -----------------------------------

#Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 0.0.0.0

#Set a custom port for HTTP:

#http.port: 9200

#For more information, consult the network module documentation.

#--------------------------------- Discovery ----------------------------------

#Pass an initial list of hosts to perform discovery when this node is started:
#The default list of hosts is [“127.0.0.1”, “[::1]”]

discovery.type: single-node
#discovery.seed_hosts: [“host1”, “host2”]

#Bootstrap the cluster using an initial set of cluster-manager-eligible nodes:

#cluster.initial_cluster_manager_nodes: [“node-1”, “node-2”]

#For more information, consult the discovery and cluster formation module documentation.

#---------------------------------- Gateway -----------------------------------

#Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

#For more information, consult the gateway module documentation.

#---------------------------------- Various -----------------------------------

#Require explicit names when deleting indices:

#action.destructive_requires_name: true
plugins.security.disabled: false
######## Start OpenSearch Security Demo Configuration ########
#WARNING: revise all the lines below before you go into production
plugins.security.ssl.transport.pemcert_filepath: /home/aiml/opensearch-2.3.0/config/node1-2.pem
plugins.security.ssl.transport.pemkey_filepath: /home/aiml/opensearch-2.3.0/config/node1-key-2.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /home/aiml/opensearch-2.3.0/config/root-ca-2.pem
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /home/aiml/opensearch-2.3.0/config/node1-2.pem
plugins.security.ssl.http.pemkey_filepath: /home/aiml/opensearch-2.3.0/config/node1-key-2.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /home/aiml/opensearch-2.3.0/config/root-ca-2.pem
plugins.security.allow_unsafe_democertificates: true
plugins.security.allow_default_init_securityindex: true
plugins.security.authcz.admin_dn:

  • ‘CN=XX,OU=YY,O=ZZ,L=BB,ST=EE,C=II’
    plugins.security.nodes_dn:
  • ‘CN=XX,OU=YY,O=ZZ,L=BB,ST=EE,C=II’
    plugins.security.audit.type: internal_opensearch
    plugins.security.enable_snapshot_restore_privilege: true
    plugins.security.check_snapshot_restore_write_privileges: true
    plugins.security.restapi.roles_enabled: [“all_access”, “security_rest_api_access”]
    plugins.security.system_indices.enabled: true
    plugins.security.system_indices.indices: [“.plugins-ml-model”, “.plugins-ml-task”, “.opendistro-alerting-config”, “.opendistro-alerting-alert*”, “.opendistro-anomaly-results*”, “.opendistro-anomaly-detector*”, “.opendistro-anomaly-checkpoints”, “.opendistro-anomaly-detection-state”, “.opendistro-reports-", ".opensearch-notifications-”, “.opensearch-notebooks”, “.opensearch-observability”, “.opendistro-asynchronous-search-response*”, “.replication-metadata-store”]
    node.max_local_storage_nodes: 3
    ######## End OpenSearch Security Demo Configuration ########

@Bindu, your OS uses a different RootCA (root-ca-2.pem) than the one defined in the opensearch.ssl.certificateAuthorities:.

Try pointing opensearch.ssl.certificateAuthorities: to root-ca-2.pem.

Hi @Pablo, we are pointing to root-ca-2.pem only. This is the path:

opensearch.ssl.certificateAuthorities: [ "/home/aiml/opensearch-2.3.0/config/root-ca-2.pem" ]

Troubleshooting of this case continued in Enabled JWT Authentication for Opensearch

Did you solve this “self signed certificate” problem? I encountered the same issue, does anyone know how to solve this?