Maintain empty shards and keep ISM policy up to date

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch Version - v 2.11.0

Describe the issue:

Handle shard size: We have set the ISM policy to rollover the index when the size goes above 30GB in order to manage the shard size. However, new policies are not implemented, and low-rate indexed shards (including empty shards) do not get rolled over very often.

We have implemented a strategy to rollover the index based on the shard size and index age. In that scenario, when the index rolls over depending on the age as well, the number of empty shards is also rising. It may exceed the maximum amount of shards per node and could have an impact on shard allocation.

But here’s where we encountered a new issue. ISM policy updates have no effect on low-size shards or empty shards unless those indexes are rolled over on purpose.

Our intention is to decrease the amount of vacant shards and rollover the index when it reaches 30GB; however, all indexes will automatically update to reflect the new policy.

Configuration:

        {
          "name": "hot",
          "actions": [
            {
              "rollover": {
                "min_index_age": "14d",
                "min_size": "90gb",
                "min_primary_shard_size": "30gb"
              },
              "retry" : {
                "count" : 100,
                "backoff" : "exponential",
                "delay" : "10m"
              }
            }
          ],
          "transitions": [
            {
              "state_name": "delete",
              "conditions": {
                "min_rollover_age": "7d"
              }
            }
          ]
        }

Relevant Logs or Screenshots: