I am running into few issues so requesting a guide line for the correct upgrade Path.
OS : Centos 7
Current ver OD : 1.8.0
Future ver OD : 1.11.1
Upgrade method is via repo and not manual rpm packages.
We run a distributed Stack [Dedicated Coordinator, Master, Data]. && Kibana
Question : What is the best way to upgrade from ver 1.8.0 to 1.11.1 Opendistro, without breaking the stack or running into issues (Ex. after upgrading Data node it can’t find Master as an example…! OR upgrading Kibana and being forced to deleted .kibana_1 or 2 index to make it work as an example).
My usual path for any upgrade is as follows:
Shut down kibana app
Update our non-datanodes (coordinators, masters, read , etc) one at a time.
Update each data node, wait for a majority of the shards to recover before proceeding to the next.
Once everything is on a new elasticsearch version, remove a kibana instance from the load balancer and upgrade then start kibana so it is the only node trying to migrate the kibana index.
once it completes, add it back so service is restored, then upgrade/restart all the other kibana nodes and they will eventually come back online.
if you use s3 for snapshots you have to upgrade the s3 plugin across all nodes before elasticsearch is restarted.
kibana may need some configuration adjustments.
make sure you have a policy for removing the default certificates for cluster communication.