There are a couple of things here…
#1 - You should not set the heap size to exactly 32GB. You need to stay slightly lower. 31GB is a common best practice. For background on why this is important see the following:
#2 - 12000 shards is an insane number of shards for an Elasticsearch node. 19000 is even worse. Again, for background see the following blog. In particular the Tip:
The number of shards you can hold on a node will be proportional to the amount of heap you have available, but there is no fixed limit enforced by Elasticsearch. A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600 shards, but the further below this limit you can keep it the better. This will generally help the cluster stay in good health.
I highly recommend that you revisit your sharding strategy.
- The best option is to simply add more Elasticsearch nodes, thus reducing the number of shards per node.
- If your indices have multiple shards on a single node, change to using a single shard per index.
- If you have multiple shards per index because you are using JBOD and multiple data paths, it is better to configure the disks for RAID-0 (or RAID-10 for a single-node cluster). You will get better performance and be able to reduce shard count.
- If you already have one shard per index (meaning you have a lot of indices) you need to combine multiple dataset per index. Add a field such as
datasetortypeso that you can more easily query specific subsets of the overall data.