Thanks. Just to follow up.
It did work successfully doing a full cluster shutdown / upgrade method.
I had a few good learnings along the way though.
For example:
The gateway.recover_after_nodes and other gateway.* settings are actually deprecated and also create more headaches than they prevent.
Dont try just start the masters without the data nodes. This is because the security indexes are likely on the data nodes… However it have been related to an issue with the gateway.* setting I had.