Dashboards isn't ready, even when pointed at a compatible Opensearch node

OpenSearch Dashboards
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Opensearch and Dashboards 3.2.0
Opensearch 2.19.1

All from official Debian packages

Describe the issue:

I upgraded Opensearch, then Dashboards on a server, from 2.19.1 to 3.2.0, but I’m getting the OpenSearch Dashboards server is not ready yet response when I try to access it.

I’ve confirmed that Opensearch is running and accessible with the username and password specified in opensearch_dashboards.yml. I successfully used the same template to generate a config file for a different cluster earlier this week.

The only error I can find is from Dashboards, saying

This version of OpenSearch Dashboards (v3.2.0) is incompatible with the following OpenSearch nodes in your cluster: v2.19.1

I saw that error in the other cluster I upgraded, but it did not prevent me from accessing Dashboards. strace and tcpdump suggest that Dashboards is not even attempting to connect to Opensearch. I’ve set logging.verbose: true and when I reload the login page the Dashboards log merely reports the same not ready error string that I get in my browser.

I was able to successfully purge 3.2.0 from that system and reinstall 2.19.1 and that still works with the same opensearch_dashboards.yml file that doesn’t for 3.2.0, but nothing that I’ve tried can make Dashboards 3.2.0, on this system, do anything.

2

@reshippie Do I understand this correctly, you have fully upgraded OS from 2.19.1 to 3.2.0 (and the cluster is now formed using only 3.2.0 nodes) and when you upgrades OSD to 3.2.0 you are getting this error: This version of OpenSearch Dashboards (v3.2.0) is incompatible with...

is this correct? Can you confirm how the cluster is being deployed?

3

No, I only upgraded the node that Dashboards connects to. It’s 1x 3.2.0 node and 14x 2.19.1 nodes. In my other cluster, which has the same number of nodes, that worked just fine.

I’m deploying them by having Ansible install the Debian packages and manage the config files.

4

@reshippie I can confirm that I can reproduce the behaviour, between any versions with 2.19.0-2 and 3.0.0 and higher.

I would recommend to raise an issue for this here, This could potentially be classified as a feature request.

5

I didn’t realize that compatibility between major versions wasn’t expected. The paragraph in the documentation about compatibility between versions seems to have disappeared after 3.0. Upgrading OpenSearch - OpenSearch Documentation
I swear I was able to pull it off on the first cluster I tried.

6

I guess I had my order of operations wrong. I was able to upgrade all of the data nodes, while keeping Dashboards running. It only stops working when I upgrade of the Manager nodes, but from there it’s a quick upgrade of the Dashboards server.