OpenSearch 3.2.0 is not able to join Existing OpenSearch 2.18.0 cluster while doing rolling upgrade

OpenSearch
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): Rolling Upgrade from 2.18.0 to 3.2.0

Describe the issue: New nodes are not able to find the 2.18.0 (existing cluster and not joining it

Configuration: Docker-based deployment on-prem x86 hosts

Relevant Logs or Screenshots:

[2025-09-12T11:35:24,277][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Will attempt to create index .opendistro_security and default configs if they are absent [2025-09-12T11:35:24,278][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Background init thread started. Install default config?: true [2025-09-12T11:35:24,279][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:24,280][INFO ][o.o.s.OpenSearchSecurityPlugin] [<NODE_NAME>] 0 OpenSearch Security modules loaded so far: [2025-09-12T11:35:25,280][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:26,280][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:27,281][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:28,281][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:29,281][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:30,282][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:31,282][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:32,283][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:33,283][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:34,250][WARN ][o.o.c.c.ClusterFormationFailureHelper] [<NODE_NAME>] cluster-manager not discovered yet: have discovered [{<NODE_NAME>}{<NODE_ID_1>}{<NODE_ID_2>}{<IP_ADDRESS_1>}{<IP_ADDRESS_1>:<PORT_1>}{d}{shard_indexing_pressure_enabled=true}]; discovery will continue using [<IP_ADDRESS_2>:<PORT_2>, <IP_ADDRESS_3>:<PORT_2>, <IP_ADDRESS_4>:<PORT_2>] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0 [2025-09-12T11:35:34,284][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:35,284][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:36,285][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:37,285][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:38,286][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:39,286][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:40,286][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:41,287][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:42,287][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:43,288][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:44,252][WARN ][o.o.c.c.ClusterFormationFailureHelper] [<NODE_NAME>] cluster-manager not discovered yet: have discovered [{<NODE_NAME>}{<NODE_ID_1>}{<NODE_ID_2>}{<IP_ADDRESS_1>}{<IP_ADDRESS_1>:<PORT_1>}{d}{shard_indexing_pressure_enabled=true}]; discovery will continue using [<IP_ADDRESS_2>:<PORT_2>, <IP_ADDRESS_3>:<PORT_2>, <IP_ADDRESS_4>:<PORT_2>] from hosts providers and from last-known cluster state; node term 0, last-accepted version 0 in term 0 [2025-09-12T11:35:44,288][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:45,289][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:46,289][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available … [2025-09-12T11:35:47,290][INFO ][o.o.s.c.ConfigurationRepository] [<NODE_NAME>] Wait for cluster to be available …

So we’re planning to add a blank node into the cluster with 3.2.0 since the cluster is already over-stressed. We’re using updated(of version 3.2.0) jvm.options and opensearch.yml for new node. Cluster is accessible on this host, which is verified by using the certificates of this host to curl request to the cluster.
What can be missing here?

2

@Dhruv can you elaborate on when are you getting these logs? Is this after the first node is taken down, upgraded and restarted? Can you share the opensearch.yml file for that node?

Also, you will need to upgrade to 2.19.2 first, in order to be able to run mix cluster and perform rolling upgrade, there are further details here

2 Likes
3

I want to configure OpenSearch through Helm, in which there is a values.yaml file

4

Thank you for your support and guidance.

The attached logs are from new hosts attempting to join the cluster. Since the cluster was under load, we are planning to add new hosts, and we thought it would be best to incorporate this as part of the upgrade process. I was under the impression that we could go directly to OpenSearch 3.0.0 from 2.18.

In any case, we tested version 2.19.x, and it successfully joined the cluster. Once the ongoing upgrade—including the cluster managers—is fully completed, we plan to move to OpenSearch 3.0.0.

Could you please confirm if it’s possible to upgrade directly from OpenSearch 2.19.x to 3.2.0, or if we need to first upgrade to 3.0.0 and then proceed to 3.2.0?

Thank you once again for your help and insights.

1 Like
5

@Dhruv yes can upgrade how directly from 2.19.0 to 3.2.0

2 Likes