How does ISM work with groups of indices?

Plugins Index Management
1

Hi all,

We currently use Curator to manage our Elasticsearch indexes. We’ve set up a system to allow us to define a list of index prefixes, with an allowance of hot data, warm data and a potential number of weeks worth of data to keep for each group. We then use bash to programmatically generate these rules for each prefix, essentially giving us rules on “groups” of indexes.
For example, a given ‘prefix group’ is ‘product-prod-’, with 20gb of hot data, 100gb of warm data and 26 weeks to keep. This will then create a rule that will check to see if all of the product-prod-* indexes with a hot tag exceed 20gb, and move indexes if any older indexes exceed that amount. Then it’ll check to see if the warm tagged ‘group’ of indexes exceed 100gb and delete anything that goes past that threshold. Lastly, if there are any indexes in that group over 26 weeks old (by the pattern in the index name), it’ll get rid of them.

My question is if we can emulate a similar system with ISM as we’re looking to migrate from ES to OpenSearch? I’ve played around a little with ISM using minimum doc count for the transitions but didn’t manage to arrive at a sensible conclusion. I guess what I want to know is how ISM works on index patterns?

For example here:
test-index-2022.09 has 2 docs
test-index-2022.10 has 2 docs
test-index-2022.11 has 7 docs

If I were to create a rule on test-index-*, to run a delete when it detects >10 docs - would it decide to delete test-index-2022.09 as there’d be 11 docs, or would it delete nothing as no individual index has over 10 docs?

Cheers!

2

ISM currently operates on single individual indices, not groups of indices. So in your example it would not delete any of the indices as none of them are over 10 documents (when creating the policy that matches test-index-* it would attach the policy to each of those indices as they are created and then they are managed individually).

There was a similar feature request for being able to transition and evaluate conditions based on a group of indices, but we have not picked it up yet. You could add your use case and a +1 over here: Transition based on cluster free available space · Issue #44 · opensearch-project/index-management · GitHub

3

Cheers for the speedy response :slight_smile: I’ll add my use case on there!