Thanks for the links. So really I’m looking for guidance in terms of number of snapshots / data size. Say I have 250 snapshots, each 50GB in size (or indeed x snapshots, each y GB in size), what hardware do I need?
The guidance for normal shards I believe is 20 shards per GB of heap memory. Is there any guidance for (remote) snapshots? I suspect that this guidance doesn’t exist yet as this is a new feature, I just don’t know where to start here.
How many GB of heap do people have vs. how many/how large snapshots have people managed to get working? Is there theoretically no limit to the number/size of searchable snapshots that can be queried in this manner, with performance being the only bottleneck?
That estimates 250 * 50 / 1024 = 12.20703125 TB of storage. not counting OS, etc… That a lot.
I keep my shard around 20-30 GB this depend on the index rotation strategy you going to use.
No sure, but I do know you can over shard your instance which i seen most do. Each environment is different, some want an index set per day, others set it per document value, some want to retain 60 Days with a weekly backup, and some want hourly backups AKA snapshots.
Need to come up with some sort of plan on the number of devices, then get a sum of what you may need. For example, I had 150 nodes ingesting about 5 GB a day using Syslog UDP, and another DMZ I had 24 nodes pushing 35 GB a day using GELF UDP. If i were you, perhaps set up a instance like Docker, throw some logs on it and calculate how many logs per day you’re getting for starters. This will make calculation much easier.