Needing some help with ISM Policy

[jbates5873]

Describe the issue:
Hi All, im just needing a little help with implementing a policy that manages the lifecycle of the indexes in our cluster.

We need to keep the data in our opensearch for 13 months for reporting and back-searching purposes.

After this period, we would like it to automatically be deleted.

We have our indexes configured to be split by month, so that we can just remove an entire index where needed, as my research indicates this was a faster and cleaner way to remove data we no longer need. rather than put it all into 1 index, and then we need to query it for the documents and remove them that way.

Currently i dont have it applied to any indexes, but my understanding is that i can input the index patterns that we use in our index templates, and then from that point forward, the templates automatically apply the ISM policy. Is this correct?

And how would I apply the policy retrospectivley to existing indexes? we currently have about 300 indexes. The main reason for implementing this first up is to control the limit we just hit of to many open shards (3000). As we have a 3 node cluster with each index configured to have 3 primary and 2 replica shards. Allowing for full parallel searching and redundancy of the data. And then every month, 4 indexes get created per user. So we want to get on top of managing the data retention. I manually removed some old indexes before that had data we no longer needed as it was from 2022, and that has given us some breathing room, but we need to work out some other things first (like automatic management of indexes) and then possibly closing off old ones at this stage, I am not sure.

Configuration:

{
    "id": "delete_indexes_older_than_1_year",
    "seqNo": 2,
    "primaryTerm": 1,
    "policy": {
        "policy_id": "delete_indexes_older_than_1_year",
        "description": "Deletes indexes that are over 400 days old.",
        "last_updated_time": 1706753812902,
        "schema_version": 18,
        "error_notification": null,
        "default_state": "delete_index",
        "states": [
            {
                "name": "default_hot_state",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "delete_index",
                        "conditions": {
                            "min_index_age": "400d"
                        }
                    }
                ]
            },
            {
                "name": "delete_index",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": []
    }
}

Relevant Logs or Screenshots:

[Gsmitt]

hey @jbates5873

I noticed in that first pic it shows “no index pattern found”. To give you a better understand here mine but its only three days.

ISM Policy 3 Day Retention

{    
    "policy": {
        "policy_id": "Winlogbeat 3 retention",
        "description": " rollover policy.",
        "last_updated_time": 1691722505995,
        "schema_version": 18,
        "error_notification": {
            "channel": {
                "id": "qR9mVYgBRnO25hGO-KYg"
            },
            "message_template": {
                "source": "",
                "lang": "mustache"
            }
        },
        "default_state": "current",
        "states": [
            {
                "name": "current",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "open": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "3d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "winlogbeat-*"
                    
                ],
                "priority": 200,
                "last_updated_time": 1681780567047
            }
        ]
    }
}

So couple things may need to be configured, this is also shown above.

1. Transitions → Here
2. index_patterns → Here

If the index pattern/s are correct it should pick up the index set and set the policy for you.

[jbates5873]

Thanks,

I intentionally left index patterns out, as i didnt want to apply it without being sure and losing prod data.

But it looks like im on the right track for configuring it then.

Thanks.