Ssl3_get_record:wrong version error in opensearch-dashboards pod

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch: latest
opensearch-dashboards: latest

Describe the issue:
I am running opensearch and opensearch-dashboards from the respective dockercompose files.

The opensearch nodes have started all right. No errors in the pod logs.
In the opensearch-dashboard pods, I am however noticing errors:
{"type":"log","@timestamp":"2023-08-07T15:14:38Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: write EPROTO 140501819639680:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:\n"}

I have disabled security plugin in the docker-compose file itself for dashboard:

opensearch-dashboards:
    image: 240852588578.dkr.ecr.us-west-2.amazonaws.com/opensearch:opensearch-dashboards
    container_name: opensearch-dashboards
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200","https://opensearch-node3:9200"]'
      DISABLE_SECURITY_DASHBOARDS_PLUGIN: true
    networks:
      - opensearch-net

Has anyone encountered similar issue before? What I am missing?

Configuration:

Relevant Logs or Screenshots:

debashish.ghosh@debashi-ltmvxkx opensearch-kube % kubectl logs opensearch-dashboards-5fb9488b9d-486pv

Disabling OpenSearch Security Dashboards Plugin

Removing securityDashboards...

Plugin removal complete

{"type":"log","@timestamp":"2023-08-07T15:13:25Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"dataSourceManagement\" has been disabled since the following direct or transitive dependencies are missing or disabled: [dataSource]"}

{"type":"log","@timestamp":"2023-08-07T15:13:25Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"dataSource\" is disabled."}

{"type":"log","@timestamp":"2023-08-07T15:13:25Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"visTypeXy\" is disabled."}

{"type":"log","@timestamp":"2023-08-07T15:13:25Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"opensearch.requestHeadersWhitelist\" is deprecated and has been replaced by \"opensearch.requestHeadersAllowlist\""}

{"type":"log","@timestamp":"2023-08-07T15:13:25Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [48] plugins: [securityAnalyticsDashboards,alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,reportsDashboards,indexManagementDashboards,anomalyDetectionDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeTable,visTypeMarkdown,visBuilder,tileMap,regionMap,customImportMapDashboards,inputControlVis,ganttChartDashboards,visualize,searchRelevanceDashboards,queryWorkbenchDashboards,notificationsDashboards,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,observabilityDashboards,discover,savedObjectsManagement,bfetch]"}

{"type":"log","@timestamp":"2023-08-07T15:13:26Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}

{"type":"log","@timestamp":"2023-08-07T15:13:30Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.20.66.64:9200"}

{"type":"log","@timestamp":"2023-08-07T15:13:30Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}

{"type":"log","@timestamp":"2023-08-07T15:13:35Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.20.244.102:9200"}

{"type":"log","@timestamp":"2023-08-07T15:13:40Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.20.132.204:9200"}

{"type":"log","@timestamp":"2023-08-07T15:13:45Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: connect ECONNREFUSED 172.20.66.64:9200"}
2

@debashish.ghosh Could you share the full docker-compose.yml file? Please also share opensearch.yml file.

3

I am also getting the same issue while installing the latest version of opensearch and opensearch dashboard 2.11.0 . Below is my opensearch.yml and opensearch_dashboard.yml

opensearch.yml

cluster.name: opensearch-cluster

# Bind to all interfaces because we don't know what IP address Docker will assign to us.
network.host: 0.0.0.0

# Setting network.host to a non-loopback address enables the annoying bootstrap checks. "Single-node" mode disables them again.
# Implicitly done if ".singleNode" is set to "true".
# discovery.type: single-node

# Start OpenSearch Security Demo Configuration
# WARNING: revise all the lines below before you go into production
plugins:
  security:
    ssl:
      transport:
        pemcert_filepath: esnode.pem
        pemkey_filepath: esnode-key.pem
        pemtrustedcas_filepath: root-ca.pem
        enforce_hostname_verification: false
      http:
        enabled: false
        pemcert_filepath: esnode.pem
        pemkey_filepath: esnode-key.pem
        pemtrustedcas_filepath: root-ca.pem
    allow_unsafe_democertificates: true
    allow_default_init_securityindex: true
    authcz:
      admin_dn:
        - CN=kirk,OU=client,O=client,L=test,C=de
    audit.type: internal_opensearch
    enable_snapshot_restore_privilege: true
    check_snapshot_restore_write_privileges: true
    restapi:
      roles_enabled: ["all_access", "security_rest_api_access"]
    system_indices:
      enabled: true
      indices:
        [
          ".opendistro-alerting-config",
          ".opendistro-alerting-alert*",
          ".opendistro-anomaly-results*",
          ".opendistro-anomaly-detector*",
          ".opendistro-anomaly-checkpoints",
          ".opendistro-anomaly-detection-state",
          ".opendistro-reports-*",
          ".opendistro-notifications-*",
          ".opendistro-notebooks",
          ".opendistro-asynchronous-search-response*",
        ]
######## End OpenSearch Security Demo Configuration ########```

opensearch_dashboard.yml

opensearch.hosts: [http://opensearch-cluster-master:9200]
opensearch.ssl.verificationMode: none
opensearch.username: admin
opensearch.password: admin
opensearch.requestHeadersWhitelist: [authorization, securitytenant]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
#opensearch_security.cookie.secure: false
server.host: '0.0.0.0'
server.basePath: "/kibana"
server.ssl.enabled: false

output of curl command

[opensearch@opensearch-cluster-master-0 ~]$ curl -vuadmin:admin http://localhost:9200
*   Trying [::1]:9200...
* Connected to localhost (::1) port 9200
* Server auth using Basic with user 'admin'
> GET / HTTP/1.1
> Host: localhost:9200
> Authorization: Basic YWRtaW46YWRtaW4=
> User-Agent: curl/8.3.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 586
< 
{
  "name" : "opensearch-cluster-master-0",
  "cluster_name" : "opensearch-cluster",
  "cluster_uuid" : "pqwusc9dQpaRXrvN05qKIw",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.11.0",
    "build_type" : "tar",
    "build_hash" : "4dcad6dd1fd45b6bd91f041a041829c8687278fa",
    "build_date" : "2023-10-13T02:55:55.511945994Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}
* Connection #0 to host localhost left intact

Here is the logs of opensearch dashboard:

{"type":"log","@timestamp":"2023-11-06T14:31:04Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
{"type":"log","@timestamp":"2023-11-06T14:31:05Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: write EPROTO C0A7861E177F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:355:\n"}
{"type":"log","@timestamp":"2023-11-06T14:31:05Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}
{"type":"log","@timestamp":"2023-11-06T14:31:07Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: write EPROTO C0A7861E177F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:355:\n"}
{"type":"log","@timestamp":"2023-11-06T14:31:09Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: write EPROTO C0A7861E177F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:355:\n"}
{"type":"log","@timestamp":"2023-11-06T14:31:12Z","tags":["error","opensearch","data"],"pid":1,"message":"[ConnectionError]: write EPROTO C0A7861E177F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:355:\n"}```

Thanks in advance
4

This is your issue.

These lines overwrite the below setting in opensearch_dashboards.yml.

As a result, OpenSearch Dashboards tries to connect to OpenSearch nodes with HTTPS instead of HTTP.