when i run hot_threads api i am seeing tons of threads generating snapshots. So i believe this is because of the automated hourly snapshot.
i have a 48 hot node (i3.2xlarge) cluster, which is indexing heavy. indices are fairly evenly balanced. peak cpu on any node is around 50-60%, average cpu for the whole cluster is even lower most of the time. but we see hourly cpu spikes that lasts few mins when avg and peak cpu goes to 100% and causes indexing back pressure to my logstash causing backup on logstash nodes. it gets cleared once the cpu is back to normal.
Checking here to see if others are facing similar issue. when i contacted aws support for it they gave me two options to disable automated snahphot for the cluster by making a backend change or make it daily snapshot vs hourly. my argument was that if we can tune the snapshot process with some limits so that it takes a little longer maybe 5 minsor so, but cause less overall peak cpu load then i should be good with it. It saves us from backlogging the logstash ingestion process because of it.