We currently use Curator to manage our Elasticsearch indexes. We’ve set up a system to allow us to define a list of index prefixes, with an allowance of hot data, warm data and a potential number of weeks worth of data to keep for each group. We then use bash to programmatically generate these rules for each prefix, essentially giving us rules on “groups” of indexes.
For example, a given ‘prefix group’ is ‘product-prod-’, with 20gb of hot data, 100gb of warm data and 26 weeks to keep. This will then create a rule that will check to see if all of the product-prod-* indexes with a hot tag exceed 20gb, and move indexes if any older indexes exceed that amount. Then it’ll check to see if the warm tagged ‘group’ of indexes exceed 100gb and delete anything that goes past that threshold. Lastly, if there are any indexes in that group over 26 weeks old (by the pattern in the index name), it’ll get rid of them.
My question is if we can emulate a similar system with ISM as we’re looking to migrate from ES to OpenSearch? I’ve played around a little with ISM using minimum doc count for the transitions but didn’t manage to arrive at a sensible conclusion. I guess what I want to know is how ISM works on index patterns?
For example here:
test-index-2022.09 has 2 docs
test-index-2022.10 has 2 docs
test-index-2022.11 has 7 docs
If I were to create a rule on test-index-*, to run a delete when it detects >10 docs - would it decide to delete test-index-2022.09 as there’d be 11 docs, or would it delete nothing as no individual index has over 10 docs?