Strategies for Crawling and Managing Outdated Indexes in OpenSearch

Plugins Index Management
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 1.2

Describe the issue:
I apologize if my questions seem basic. As far as I’ve researched, it seems that OpenSearch only allows the creation or deletion of documents from an index, without providing a direct way to track when the index was last updated. However, I’m interested in knowing what strategy the community is currently using to determine if an index hasn’t been updated for some time.
Currently, my client can request a new indexing of records based on my implementation, which results in the creation of a new index, dissociating the previous index from their environment. I believe that handling the deletion of the previous index, which is no longer in use, could be done using ISM.

Configuration:
{
“cluster_name”: “130210428544:opensearch-12-us”,
“status”: “yellow”,
“timed_out”: true,
“number_of_nodes”: 1,
“number_of_data_nodes”: 1,
“discovered_master”: true,
“active_primary_shards”: 763,
“active_shards”: 763,
“relocating_shards”: 0,
“initializing_shards”: 0,
“unassigned_shards”: 70,
“delayed_unassigned_shards”: 0,
“number_of_pending_tasks”: 0,
“number_of_in_flight_fetch”: 0,
“task_max_waiting_in_queue_millis”: 0,
“active_shards_percent_as_number”: 91.59663865546219
}

Relevant Logs or Screenshots:
N/A

2

Hey @johnenderson

Have you created a ISM policy? That should rotate, delete, etc… old index sets.

Through ISM you can do much more which is found here.

This can be done by making a channel , then attaching it to the ISM policy which is found here Error notifications.

EDIT: ISM policy, in the action section is a notification configuration.

image
image1348×633 43.1 KB

3

Yes, I have a defined policy. Upon reviewing the settings, I noticed that I only found the parameter min_index_age, which seems to refer to the minimum age of the index. However, I’m not interested in viewing just the minimum age of the index, as I may have indices with that minimum age still being used by my clients. My intention is to identify the indices that haven’t been updated for more than 60 days.

{
    "id": "delete-after-60-days",
    "seqNo": 176541,
    "primaryTerm": 7,
    "policy": {
        "policy_id": "delete-after-60-days",
        "description": "Policy to delete indices after 60 days of inactivity.",
        "last_updated_time": 1702518191502,
        "schema_version": 13,
        "error_notification": {
            "destination": {
                "custom_webhook": {
                    "path": null,
                    "header_params": {
                        "Content-Type": "application/json"
                    },
                    "password": null,
                    "scheme": null,
                    "port": -1,
                    "query_params": {},
                    "host": null,
                    "url": "XYZ",
                    "username": null
                }
            },
            "message_template": {
                "source": "The index {{ctx.index}} failed during policy execution.",
                "lang": "mustache"
            }
        },
        "default_state": "init",
        "states": [
            {
                "name": "init",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "60d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "notification": {
                            "destination": {
                                "custom_webhook": {
                                    "path": null,
                                    "header_params": {
                                        "Content-Type": "application/json"
                                    },
                                    "password": null,
                                    "scheme": null,
                                    "port": -1,
                                    "query_params": {},
                                    "host": null,
                                    "url": "XYZ",
                                    "username": null
                                }
                            },
                            "message_template": {
                                "source": {
                                    "@context": "https://schema.org/extensions",
                                    "@type": "MessageCard",
                                    "summary": "OpenSearch Posted",
                                    "themeColor": "0072C6",
                                    "title": "Amazon OpenSearch Service Posted",
                                    "sections": [
                                        {
                                            "startGroup": true,
                                            "title": "**Exclusion Policy: 60 Days Inactivity**",
                                            "text": "This index has been removed after 60 days of inactivity, as per policy",
                                            "facts": [
                                                {
                                                    "name": "Index name:",
                                                    "value": "{{ctx.index}}"
                                                },
                                                {
                                                    "name": "Index UUID:",
                                                    "value": "{{ctx.index_uuid}}"
                                                },
                                                {
                                                    "name": "Policy name:",
                                                    "value": "{{ctx.policy_id}}"
                                                },
                                                {
                                                    "name": "Region:",
                                                    "value": "us-east-1"
                                                },
                                                {
                                                    "name": "OpenSearch Version",
                                                    "value": "v1.2.0"
                                                }
                                            ]
                                        }
                                    ]
                                },
                                "lang": "mustache",
                                "options": {
                                    "content_type": "application/json; charset=UTF-8"
                                }
                            }
                        }
                    },
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "host*"
                ],
                "priority": 1,
                "last_updated_time": 1699923209719
            }
        ]
    }
}
4

@Gsmitt , help?

5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.