Currently we have a cluster with 41TB of data with around ~ 9100 shards with 12 nodes 64GB each.
All nodes have all roles right now.
We already know we are oversharded and we are running way over the recommended 30:1 ratio of data to memory.
Now we had an issue which brought our cluster down, and now I’m seeing nodes which are complaining about time outs while talking to the master so my current reasoning is that master is doing too much so a master-only node looks like a good option.
Then I arrive in my current issue: I didn’t found any documentation regarding master-only sizing / capacity planning. How much memory a master-only node needs? What’s the recommended quantity of memory?
I’m thinking of adding 3 master-only nodes. How much memory I need in each node? I’m guessing by the lack of cpu usage in master I guess this is not a cpu heavy role?
TL;DR: I need guides / directions regarding master-only capacity planning / sizing.