ISM policy not deleting the old indices

Plugins Index Management
1

Following is the index policy I have created

    "policy": {
        "policy_id": "delete_after_2_days",
        "description": "Policy that deletes indicies older than 2days.",
        "last_updated_time": 1613056975480,
        "schema_version": 1,
        "error_notification": null,
        "default_state": "open",
        "states": [
            {
                "name": "open",
                "actions": [
                    {
                        "open": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "2d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ]
    }
}

Then I set the template as

PUT _template/delete-indices-after-2days
{
    "index_patterns": [
        "test-index-*"
    ],
    "settings": {
        "index.opendistro.index_state_management.policy_id": "delete_after_2_days"
    }
}

After that, I applied this to old indices like below

PUT test-index*/_settings
{
  "settings": {
    "index.opendistro.index_state_management.policy_id": "delete_after_2_days"
  }
}

But nothing seems to have happened. I can see older indices of test-index.
What could be possibly wrong?

2

Hey @curiousmind,

So the template should be applying to any newly creating indices. And then you manually applied to old with API. When you say nothing seems to have happened do you mean no managed indices were created at all? Or they just aren’t running? Are you on Amazon Elasticsearch Service or self-hosted ODFE?

Thanks

3

I am on self-hosted ODFE.
No after the above steps, the old indices like (test-index-2020-02-07, test-index-2020-02-08, which are more than 2 days old) should be deleted right?. I dont see that happening. Or will it take time. I applied those changes yesterday when I wrote the question.

4

Yes, if everything went as planned then those indices should have been deleted. Trying to understand where it failed as it could be multiple places.

  1. What should have happened after you applied the policy to those indices is a managed index job was created. You could check on Kibana - Index Management - Managed Indices page and see if you see two for those indices. Or you can directly query the .opendistro-ism-config index and see if you can see two for those indices. Can you check for these (or you can try #2 below first)?
  2. After the job is created it is scheduled and executed. The first execution should initialize the metadata. You can check if metadata was initialized by calling the _opendistro/_ism/explain/test-index*?pretty API and pasting the response for those two indices. If there is information about the policy/state/action/or transition on there then it was successfully initialized. If it only shows the opendistro.index_state_management.policy_id setting then it hasn’t been initialized. Initializing metadata happens after job creation so if you see it has initialized then #1 was successful.
  3. It had to execute open. Did this fail? If so it would show in the explain API response.
  4. It had to transition to delete after checking index age of 2d, you can check the index age yourself by calling /_settings and looking for the creation date. I’m assuming the indices are in fact over 2 days based on name.
  5. Did it fail delete? Would show in explain API.

So would like you to check if job was created, if explain shows a response, and if it does why did it fail? Or does it say it’s running something?

5

@dbbaughe I have facing problem with ILM snapshot policy job creation for indexes. The policy was applied to the indices(all the indices are green and no red) successfully.
However, i don’t see any movement in state/action as it says, “Still initializing, please wait”. I waited for more than 2 days and this status remains the same.
As i read the point #2 from your comment, I executed _opendistro/_ism/explain/test-index*?pretty and i get only the “index.opendistro.index_state_management.policy_id” and index name. I am not sure it is failing at job creation of metadata initialization.
I have attached the Managed indices snapshot. I appreciate if you could help me in fixing this issue :slight_smile:

ILM_Snapshot
ILM_Snapshot1901×707 61.7 KB

6

Hey @bala,

Do you have access to the elasticsearch logs? Can you remove and re-apply to start the process over again and then look at the logs after ~10 minutes (should have executed by then as on ODFE/OS it defaults to 5 minute intervals I believe). Looks like you do have a job created if it’s showing up on the UI. So if it’s failing to initialize I would expect maybe something wrong with either the job scheduling, loading/saving the policy to the job, or initializing the metadata.