Describe the issue: When we try to rollback any minor version upgrades especially with lucene version difference like in 2.11 to 2.15 where all data nodes are upgraded to 2.15 and we decided to rollback to 2.11 again here while rollback each node shards are not getting allocated due to replica must be greater or equal to primary version. Here what was the solution to safety rollback the upgraded nodes without traffic intruption
Configuration: 4 data nodes 3 master nodes initially at 2.11 and 4 data nodes rolling upgraded to 2.15 then we dediced to rollback these 4 nodes back to 2.11
Relevant Logs or Screenshots:
cannot allocate replica shard to a node with version 2.11.0 since this is older than the primary version 2.15.0
OpenSearch does not support downgrading nodes, including between minor versions. The shard allocation error you’re seeing is consistent with the version compatibility restriction, as a replica shard cannot be allocated to a node running an older version than its primary.
According to the official rolling upgrade documentation: https://docs.opensearch.org/latest/migrate-or-upgrade/rolling-upgrade/
the documented rollback procedure requires a fresh installation of the previous version and restoration from a compatible snapshot.
If you have a compatible pre-upgrade snapshot, you could deploy a separate OpenSearch 2.11 cluster, restore the snapshot, and validate it before switching traffic. However, any data written after that snapshot would need to be handled separately to avoid data loss.
Do you have a snapshot taken before upgrading to 2.15?