Index Management question

I’m trying to set up a policy which deletes indexes after 3 days. I’ve looked at the documentation here but I’m still getting an error when trying to apply the policy.

Below is my IM policy I set up.

{
    "policy": {
        "policy_id": "common-lateattach-delete",
        "description": "open warm delete workflow",
        "last_updated_time": 1605293466946,
        "schema_version": 1,
        "error_notification": null,
        "default_state": "open",
        "states": [
            {
                "name": "open",
                "actions": [
                    {
                        "rollover": {
                            "min_index_age": "1d"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "warm"
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "replica_count": {
                            "number_of_replicas": 5
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "3d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "notification": {
                            "destination": {
                                "chime": {
                                    "url": "https://hooks.slack.com/services/##########"
                                }
                            },
                            "message_template": {
                                "source": "The index {{ctx.index}} is being deleted",
                                "lang": "mustache"
                            }
                        }
                    },
                    {
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ]
    }
}

Below is the template I have applied to my indexes.

{
  "lateattach_common" : {
    "order" : 0,
    "index_patterns" : [
      "lateattach-common-*"
    ],
    "settings" : {
      "index" : {
        "opendistro" : {
          "index_state_management" : {
            "policy_id" : "common-lateattach-delete"
          }
        }
      }
    },
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        }
      }
    },
    "aliases" : {
      "live" : { }
    }
  }
}

When the IM policy tries to implement a change, I see the following error.

 "info" : {
      "message" : "Missing rollover_alias index setting [index=lateattach-common-2020.11.16]"
    }

What am I missing in my policy that is preventing it from doing the rollover and then deleting the index?

Hey @jberto78,

Rollover operates on an alias , rather than an index. And because an index can have multiple aliases, you need to provide ISM enough information to know which alias it should be rolling over.

The way to do that is through another index level setting
"opendistro.index_state_management.rollover_alias": "some_alias". You can add this to your index template in the same place as the policy_id one. This should be the alias that you want to be rolled over, so every new index will automatically have this setting and know this is the alias to actually rollover. When you create your first index for this index group, you will need to bootstrap it with the alias you configured.

Hi @dbbaughe,
I did as you recommended and initially it appears to have worked (I didn’t get the “missing rollover_alias” message). But now I have the below error when the policy tries to rollover the index (see below). I do see that the policy is trying to connect to our instance on port 9300 but since we’re running on AWS Elasticsearch I don’t have that port open. Can that be configured to use a different port?

{
    "cause": "[c94c772aaf871a37ca3be1658beb9435][x.x.x.x:9300][indices:admin/rollover]",
    "message": "Failed to rollover index [index=index_name-2020.11.18]"
}

Does the user that is running the policy have the rights (admin/rollover) on the index?

I don’t have it set to run as a specific user, so I guess it would be under the admin user? Is there a default user that Index Management runs under when trying to rollover or delete an index? Is there a way I can check what user the script ran under?

Hey @jberto78,

Could you cut a support ticket for AWS, there was a bug with the handling of exceptions in rollover specifically which is why you’re not seeing the actual error. Need support to check out the logs in elasticsearch.log to tell you explicitly why it failed which will be a bit better than guessing.