Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations

Error

Today when I try to redeploy OpenSearch Dashboards, I got this error

{"type":"log","@timestamp":"2022-06-15T17:20:36Z","tags":["info","plugins-service"],"pid":1,"message":"Plugin \"visTypeXy\" is disabled."}
{"type":"log","@timestamp":"2022-06-15T17:20:36Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpu.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuPath\""}
{"type":"log","@timestamp":"2022-06-15T17:20:36Z","tags":["warning","config","deprecation"],"pid":1,"message":"\"cpuacct.cgroup.path.override\" is deprecated and has been replaced by \"ops.cGroupOverrides.cpuAcctPath\""}
{"type":"log","@timestamp":"2022-06-15T17:20:36Z","tags":["info","plugins-system"],"pid":1,"message":"Setting up [44] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,reportsDashboards,securityDashboards,indexManagementDashboards,anomalyDetectionDashboards,dashboard,visualizations,visTypeVega,visTypeTimeline,timeline,visTypeMarkdown,visTypeTable,tileMap,regionMap,inputControlVis,ganttChartDashboards,visualize,queryWorkbenchDashboards,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,observabilityDashboards,discover,savedObjectsManagement,bfetch]"}
{"type":"log","@timestamp":"2022-06-15T17:20:36Z","tags":["info","savedobjects-service"],"pid":1,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
{"type":"log","@timestamp":"2022-06-15T17:20:37Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:20:37Z","tags":["error","savedobjects-service"],"pid":1,"message":"Unable to retrieve version information from OpenSearch nodes."}
{"type":"log","@timestamp":"2022-06-15T17:20:39Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:20:42Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:20:44Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:20:47Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:20:49Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:20:52Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:20:54Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:20:57Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:20:59Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:02Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:04Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:07Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:09Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:12Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:14Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:17Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:19Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:22Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:24Z","tags":["error","opensearch","data"],"pid":1,"message":"[ResponseError]: Response Error"}
{"type":"log","@timestamp":"2022-06-15T17:21:26Z","tags":["info","plugins-system"],"pid":1,"message":"Stopping all plugins."}
{"type":"log","@timestamp":"2022-06-15T17:21:26Z","tags":["info","savedobjects-service"],"pid":1,"message":"Starting saved objects migrations"}
{"type":"log","@timestamp":"2022-06-15T17:21:26Z","tags":["warning","savedobjects-service"],"pid":1,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."} 

The only environment I passed as Kubernetes secret/environment is

OPENSEARCH_HOSTS=https://xxx.com:9200

And I can succeed deploying before.

When I GET https://xxx.com:9200 in Postman, I can see

{
    "name": "xxx",
    "cluster_name": "xxx",
    "cluster_uuid": "xxx",
    "version": {
        "distribution": "opensearch",
        "number": "1.2.4",
        "build_type": "tar",
        "build_hash": "e505b10357c03ae8d26d675172402f2f2144ef0f",
        "build_date": "2022-01-14T03:38:06.881862Z",
        "build_snapshot": false,
        "lucene_version": "8.10.1",
        "minimum_wire_compatibility_version": "6.8.0",
        "minimum_index_compatibility_version": "6.0.0-beta1"
    },
    "tagline": "The OpenSearch Project: https://opensearch.org/"
}

OpenSearch Dashboards Version

I am using OpenSearch Dashboards 1.2.0. Because currently, our OpenSearch is using 1.2.4. OpenSearch Dashboards 1.2.0 is compatible with OpenSearch 1.2.4. OpenSearch Dashboards 1.3.0 is NOT compatible with OpenSearch 1.2.4.

opensearch_dashboards.yml

For the container running, it is

sh-4.2$ cat opensearch_dashboards.yml
---
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

# Description:
# Default configuration for OpenSearch Dashboards

server.host: '0'
opensearch.hosts: [https://localhost:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersWhitelist: [authorization, securitytenant]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

I think the opensearch.hosts does not matter in this case, as the environment I set in the Kubernetes environment will overwrite it.

I cannot ssh into the new container I try to redeploy because it is CrashLoopBackOff all time.

I found some similar tickets at

But not have a solution. Any guide would be helpful!

I found the issue!

One of my coworker deleted the default user kibanaserver for OpenSearch for security concern.

If you see opensearch_dashboards.yml above, by default, it is using kibanaserver.

---
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0

# Description:
# Default configuration for OpenSearch Dashboards

server.host: '0'
opensearch.hosts: [https://localhost:9200]
opensearch.ssl.verificationMode: none
opensearch.username: kibanaserver
opensearch.password: kibanaserver
opensearch.requestHeadersWhitelist: [authorization, securitytenant]

opensearch_security.multitenancy.enabled: true
opensearch_security.multitenancy.tenants.preferred: [Private, Global]
opensearch_security.readonly_mode.roles: [kibana_read_only]
# Use this setting if you are running opensearch-dashboards without https
opensearch_security.cookie.secure: false

Based on the document at Docker - OpenSearch documentation

If you override opensearch_dashboards.yml settings using environment variables, as seen above, use all uppercase letters and underscores in place of periods (e.g. for opensearch.hosts , use OPENSEARCH_HOSTS ).

To give OpenSearch Dashboards correct username and password to connect OpenSearch.
I just need set the right username and password by passing

OPENSEARCH_USERNAME=xxx
OPENSEARCH_PASSWORD=xxx

as Kubernetes environment that OpenSearch is using.

i am not able to configure my default password in opensearch dashboard . I install using docker-compose file. please troublleshoot my error.