ISM shrink issue

Plugins Index Management
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

2.8.0

Describe the issue:

Hello, everyone. I’m having trouble understanding how ISM works. I have the following configuration with which there are misunderstandings. If I understand correctly, when Shrink occurs, a new index is created and the old one is deleted. Accordingly, subsequent policy actions will not be applied to the new index. That is, the Force merge and move to cold actions will not occur. Also, I don’t understand how to add aliases during Shrink as they were in the original index. And lastly, maybe you can share your ISM examples so that I can create my own based on them. Thank you very much.

Configuration:

{
    "policy": {
        "policy_id": "logs-mssql.service-prod",
        "description": "Windows MS SQL error logs",
        "last_updated_time": 1690445583171,
        "schema_version": 18,
        "error_notification": null,
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "rollover": {
                            "min_size": "120gb",
                            "min_index_age": "30d"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "warm",
                        "conditions": {
                            "min_rollover_age": "1d"
                        }
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "allocation": {
                            "require": {
                                "temp": "warm"
                            },
                            "include": {},
                            "exclude": {},
                            "wait_for": false
                        }
                    },
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "shrink": {
                            "num_new_shards": 3,
                            "target_index_name_template": {
                                "source": "{{ctx.index}}_shrunken",
                                "lang": "mustache"
                            },
                            "force_unsafe": false
                        }
                    },
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "force_merge": {
                            "max_num_segments": 1
                        }
                    },
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "rollover": {
                            "min_index_age": "120d"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "cold",
                        "conditions": {
                            "min_rollover_age": "1d"
                        }
                    }
                ]
            },
            {
                "name": "cold",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "allocation": {
                            "require": {
                                "temp": "cold"
                            },
                            "include": {},
                            "exclude": {},
                            "wait_for": false
                        }
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "logs-mssql.service-prod"
                ],
                "priority": 1,
                "last_updated_time": 1687519653008
            }
        ]
    }
}

Relevant Logs or Screenshots:

2

@Eugene7 Perhaps you can help me?

3

Hey @Kruzerson

I haven thad that issue but my index patterns are like this.

"ism_template": [
            {
                "index_patterns": [
                    "winlogbeat", <----------- my_index_alias
                    "winlogbeat-*"                   
                ],
                "priority": 200,
                "last_updated_time": 1681780567047
            }
        ]

Prior to send logs or data to Opensearch, I created a index template /w alias .

image
image1199×262 14.5 KB

When i want to shink, delete or migrating, I do not have an issue. Maybe im just lucky.

4

Hello @Gsmitt . I tried adding alias to ISM, I’ll see what happens. Can you tell me, after the index has become a shrink, can you see this index in the discovery?

5

Hey @Kruzerson

When I get a chance I will.

6

Thanks @Gsmitt. Also, after the shrink, how do you manage the new index?

7

I have the same question. Do we need to make a “shrunken” ISM policy that handles these new shrunken indexes? i.e. deletes them after a set period of time? And then in the original ISM policy, immediately delete the pre-shrunken index?

8

Hi @Seth.L I already answered you in Slack :slight_smile: Think about whether you need to use a shrink at all.

Yes, I’ve already realized that, too. Unfortunately, ISM is very different. The only way I’ve come up with is to add a main alias and a “log-shrink” alias when doing shrink. Then create another ISM that will manage the newly created index by the alias “log-shrink”. And then you can remove the replica and perform other actions. But these are “Quick-and-dirty solution”. So I decided to change the indexing strategy.

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.