Hi,
I’m trying to upgrade from OS 1.2.3 to OS 1.2.4 in Kubernetes environment (SLES 15).
During this upgrade, I’m shutting down data nodes one by one to make sure all shards are in assigned state before upgrading the next data pod.
Total no. of indices : 10, total no. of shards : 50 primary, 50 secondary (5 pri shards for each index and 1 replica).
Here, the cluster is going to red state saying
"allocation_explanation" : "cannot allocate because the cluster is still waiting 2.7m for the departed node holding a replica to rejoin, despite being allowed to allocate the shard to at least one other node"
"explanation" : "reached the limit of outgoing shard recoveries [2] on the node [LDLvL8FLTTSm-kxaHL4o6w] which holds the primary, cluster setting [cluster.routing.allocation.node_concurrent_outgoing_recoveries=2] (can also be set via [cluster.routing.allocation.node_concurrent_recoveries])"
Is this expected behaviour during upgrade ?