Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.5 to 2.9 rolling upgrade
Describe the issue:
Do I miss any additional permissions while upgrading from 2.5 to 2.11 ?
Stopping any master node is OK while all nodes are on 2.5, but opensearch service stop fails, for hours with below logs, during rolling upgrade on cluster manager node which is still on 2.5. Seams 2.5 cluster manager is having problems to check followers which are upgraded to 2.11. Cluster is green, and upgraded masters still see 2.5 node as cluster manager.
Configuration:
Relevant Logs or Screenshots:
{"type": "server","level": "INFO","component": "o.o.c.c.FollowersChecker","cluster.name": "c_name","node.name": "master1a.pdx2","message": "FollowerChecker{discoveryNode={master2b.pdx2}{...}{aws_availability_zone=us-west-2b, shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=1,[cluster.fault_detection.follower_check.retry_count]=3} failed, retrying", "cluster.uuid": "uuid", "node.id": "id" ,"stacktrace": ["org.opensearch.transport.SendRequestTransportException: [master2b.pdx2][IP:9300][internal:coordination/fault_detection/follower_check]","at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:941) [opensearch-2.5.0.jar:2.5.0]",