Stuck at "Attempting to rollover"

I’ve created the following policy based on the demo policy in the docs.

curl --cacert config/ca.pem -X PUT "https://localhost:9200/_opendistro/_ism/policies/hot_warm_delete" -H 'Content-Type: application/json' -d'
    {
        "policy": {
            "policy_id": "hot_warm_delete",
            "description": "hot warm delete workflow",
            "last_updated_time": 1592427149559,
            "schema_version": 1,
            "error_notification": {
                "destination": {
                    "slack": {
                        "url": "https://hooks.slack.com/services/OMITTED"
                    }
                },
                "message_template": {
                    "source": "There was an error on index {{ctx.index}}",
                    "lang": "mustache"
                }
            },
            "default_state": "hot",
            "states": [{
                    "name": "hot",
                    "actions": [{
                        "rollover": {
                            "min_index_age": "1d"
                        }
                    }],
                    "transitions": [{
                        "state_name": "warm"
                    }]
                },
                {
                    "name": "warm",
                    "actions": [{
                        "replica_count": {
                            "number_of_replicas": 5
                        }
                    }],
                    "transitions": [{
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "1d"
                        }
                    }]
                },
                {
                    "name": "delete",
                    "actions": [{
                            "notification": {
                                "destination": {
                                    "slack": {
                                        "url": "https://hooks.slack.com/services/OMITTED"
                                    }
                                },
                                "message_template": {
                                    "source": "The index {{ctx.index}} is being deleted",
                                    "lang": "mustache"
                                }
                            }
                        },
                        {
                            "delete": {}
                        }
                    ],
                    "transitions": []
                }
            ]
        }
    }
    '

I’ve also created a template which (from my understanding) will allow an index matching the template’s configured index pattern to automatically get assigned to the management policy with the template’s rollover_alias.

curl --cacert config/ca.pem -X POST "https://localhost:9200/_template/syslog" -H 'Content-Type: application/json' -d'
      {
        "aliases": {},
        "index_patterns": ["syslog-20*"],
        "settings": {
          "opendistro.index_state_management.policy_id": "hot_warm_delete",
          "opendistro.index_state_management.rollover_alias": "syslog"
        }
      }'

And then I’ve created the necessary alias:

curl --cacert config/ca.pem -X POST "https://localhost:9200/_aliases" -H 'Content-Type: application/json' -d'
    {
        "actions": [
          {
            "add": {
              "index": "syslog-20*",
              "alias": "syslog"
            }
          }
        ]
      }
      '

New indexes are created based on timestamp (ie. syslog-2020.06.24) which match the index pattern, and I can see the index being managed. It will initialize successfully, but stays in a “attempting to rollover” state.

What am I doing wrong here? I’ve scoured the forums and the docs but I can’t tell where I’m going wrong. Some help would be much appreciated! :slight_smile:

Hi @jbarnett,

I’m assuming the indices that are stuck on “Attempting to rollover” are over 1 day old right?
Which version are you on? If on one of the newer versions you should see the conditions next to the message.

Thanks @dbbaughe, actually that was part of the issue I think but one of the indexes just hit the 1d mark and now it’s showing as failed.

{
    "cause": "[OMITTED][OMITTED:9300][indices:admin/rollover]",
    "message": "Failed to rollover index"
}

@jbarnett
For this you’ll need to check the elasticsearch.log to see why it failed to rollover the index as this looks like the API call itself failed internally.

I’m not seeing anything in the logs at all

@jbarnett,

Is it a single node domain? Or if there are multiple nodes are you sure you’re checking the right node for logs?

The message you see is from here which just happens when the rollover request here fails. There should be a log somewhere.

Ah ok here we go. I’m actually running this in ECS and I have 3 instances of ES running. The logs were on one of the other instances.

Here’s the relevant error:

[2020-06-25T22:18:37,211][ERROR][c.a.o.i.s.r.AttemptRolloverStep] [OMITTED] Failed to rollover index [index=syslog-2020.06.24]
org.elasticsearch.transport.RemoteTransportException: [OMITTED][OMITTED:9300][indices:admin/rollover]
Caused by: java.lang.IllegalArgumentException: source alias [syslog] does not point to a write index

@jbarnett,

Ah got it, you added an alias to multiple indices which means rollover does not know which index to rollover. For the specific index that you want to be rolled over, you need to add is_write_index as seen here.

Hrm. Well what I have is new indexes being created daily named:
syslog-YYYY.MM.DD
logstash-YYYY.MM.DD

I’d like for these daily indexes to be picked up automatically and managed by the policy, and then rolled over per the policy. How would I achieve that?

You should be able to do that with your above settings, you just need to make sure the active index being written to has the is_write_index setting set to true so the rollover API knows which index to rollover.

So even if you end up with say 10 syslog indices that look like
syslog-2020.06.20
syslog-2020.06.21
syslog-2020.06.22
syslog-2020.06.23

syslog-2020.06.29

All of them would have is_write_index: false except for the latest one. And once it reaches 1 day it will be rolled over into your new syslog-2020.06.30.

Just to clarify on one point though:
“Hrm. Well what I have is new indexes being created daily named:

I’d like for these daily indexes to be picked up automatically and managed by the policy, and then rolled over per the policy”
The rollover should be the one creating your new indices for you at this point once you’re using ISM to rollover for you.

thanks for the clarification - this is really helpful. So one question, how would I automate this so that the first index created has “is_write_index” set to true so that I don’t need to manually set it to start this process?

I’ve created a separate index titled “syslog-000001” and associated to the syslog alias with is_write_index = true and this seems to be working now. But it’s also incrementing that index and I now have syslog-000001 - syslog-000003

Hi All,
So I am using fluentd to send the logs to elasticsearch, so the name of the indices are being configured with fluentd now how do I set the aliases in fluentd ? so, automatically indices have aliases and so i can apply policies to ISM and that could make it automatically? any suggestions?

Also would be happy to know how to automatically apply these templates to elasticsearch.

Thank you All