ISM policies for Live & Restored Indices

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.7

Describe the issue:

I have two ISM policies. Initial policy takes care of the current live index and creates a restored index with remote*. The second policy is to delete the restored indices after the retention period. The initial policy gets attached automatically when the index gets created but the 2nd policy does not. The 2nd policy gets attached to the restored indices on a random basis.

I am attaching both policies here to see if there is an issue with how I have set it up

Configuration

Initial Policy
==============
{
    "policy": {
        "policy_id": "initial-hvuv-osdev-30m-ism-policy",
        "description": "Hot 7d TO  Warm 1m TO S3 Snapshot TO Delete at 15m",
        "last_updated_time": 1781792440941,
        "schema_version": 29,
        "error_notification": {
            "channel": {
                "id": "L_3gbp4BMN_ORf_TcVFV"
            },
            "message_template": {
                "source": "ISM policy execution failed.\n\n==============\nPolicy: {{ctx.policy_id}}\nIndex: {{ctx.index}}\nState: {{ctx.state.name}}\nAction: {{ctx.action.name}}\nReason: {{ctx.info.message}}\n==============",
                "lang": "mustache"
            }
        },
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "index_priority": {
                            "priority": 100
                        }
                    },
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "rollover": {
                            "min_doc_count": 200000000,
                            "min_index_age": "7d",
                            "min_primary_shard_size": "50gb",
                            "copy_alias": false
                        }
                    },
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "force_merge": {
                            "max_num_segments": 1
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "searchable_snapshot",
                        "conditions": {
                            "min_index_age": "30s"
                        }
                    }
                ]
            },
            {
                "name": "searchable_snapshot",
                "actions": [
                    {
                        "retry": {
                            "count": 2,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "replica_count": {
                            "number_of_replicas": 0
                        }
                    },
                    {
                        "retry": {
                            "count": 1,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "snapshot": {
                            "repository": "osdev-hvuv-ism",
                            "snapshot": "osdev-14d-ism-{{ctx.index}}"
                        }
                    },
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "convert_index_to_remote": {
                            "repository": "osdev-hvuv-ism",
                            "snapshot": "osdev-14d-ism-{{ctx.index}}",
                            "include_aliases": true,
                            "ignore_index_settings": "",
                            "delete_original_index": true,
                            "rename_pattern": "remote_$1"
                        }
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "logs-*"
                ],
                "priority": 400,
                "last_updated_time": 1781792440941
            }
        ]
    }
}

Delete Policy
============
{
    "id": "vz-tsgi-hvuv-osdev-cleanup-14d-ism-policy",
    "seqNo": 3352737,
    "primaryTerm": 14,
    "policy": {
        "policy_id": "osdev-cleanup-14d-ism-policy",
        "description": "Policy to remove remote indices after 14 days",
        "last_updated_time": 1781781641723,
        "schema_version": 29,
        "error_notification": {
            "channel": {
                "id": "L_3gbp4BMN_ORf_TcVFV"
            },
            "message_template": {
                "source": "ISM policy execution failed.\n\n==============\nPolicy: {{ctx.policy_id}}\nIndex: {{ctx.index}}\nState: {{ctx.state.name}}\nAction: {{ctx.action.name}}\nReason: {{ctx.info.message}}\n==============",
                "lang": "mustache"
            }
        },
        "default_state": "intial_state",
        "states": [
            {
                "name": "intial_state",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_rollover_age": "7d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "remote_*"
                ],
                "priority": 450,
                "last_updated_time": 1781781641723
            }
        ]
    }
}

Relevant Logs or Screenshots: Not applicable.

2

@muraliv this appears to be very similar case to this forum topic. Which seems to point to a possible race condition in the scheduler described in issue 275.

How are you running this cluster, is in in k8s? Are you using operator?

3

@Anthony - No k8s involved or operator involved.

4

@Anthony - I am running into this issue.

All of our indices are datastreams and they do have that setting “index.hidden”: true based on this issue. I changed just the convert_to_index_remote action of the ISM policy like this.

"convert_index_to_remote": {
"repository": "osdev-tsgi-hvuv-ism",
"snapshot": "osdev-14d-ism-{{ctx.index}}",
"include_aliases": true,
"ignore_index_settings": "index.hidden",
"delete_original_index": true,
"rename_pattern": "remote_$1"
}

The index does not have that setting but no clean up policy attached to it. So I need to run a script to attach it to the remote* indices?

Thanks

Murali

5

@muraliv your approach to remove the hidden tag indeed works as expected, however can you check whether creation_date of the indices in question predates their cleanup policy’s last_updated_time. If last_updated_time is newer/later than the indices, they will not attach, as snapshot restore preserves the original index.creation_datemetadata of the source index, it does not stamp a fresh one at restore time.

6

@Anthony - What is the solution for this scenario ? We ended up modifying the cleanup policy for remote indices couple of times to add “error notification” and also to use the rollover age instead of the index age. There could be other potential reasons for us to update the clean up policy.

Is the solution is to delete and recreate the ISM policy and use the last_update_time to few months back ?

Thanks

Murali

7

Hi @Anthony

I have the index creation.date newer than the ISM policy last_update_time, but still the policy is not getting attached. I looked at the index settings and the index_state_management.enabled is false for these indices. I also looked at the cluster settings and it’s set to true.

What is the scenario that this setting gets changed from true to false?

I have already shared the initial policy here in this topic. I am working with two other cluster admins and they are not running into this issue, just me, the lucky one.

Thanks

Murali

8

Hi @Anthony

I found the root cause for this. This is the scenario I was running into.

When I update an ISM policy (the initial policy) , I manually remove the ISM policy using PUT plugins/_ism/remove/.ds-logs* and add the new policy using PUT plugins/_ism/add/.ds-logs*. This causes the index.plugins.index_state_management.auto_manage setting to change from the default true to false.

Hopefully this will fix the issue going forward.

Thanks

Murali