The docs for a basic roll-over seem to be lacking a bit around how the index write alias is managed.
Does it have to exist on the index or will ISM add it?
When an index “Rolls over” for various conditions, does the new index automatically get the alias and the previous index has the alias removed?
Not sure if it was my setup or what but I just tested a daily roll over and I had 5 indices all with the same alias (the one I use to write)
Any suggestions?
Adding some more weirdness to what I am seeing with the 1.7 release:
create a rollover policy:
{ "policy_id": "container-rollover", "description": "A simple default that rolls indexes over at 1GB or 1 day.", "last_updated_time": 1591139176328, "schema_version": 1, "error_notification": null, "default_state": "active", "states": [ { "name": "active", "actions": [ { "rollover": { "min_size": "1gb", "min_index_age": "1d" } } ], "transitions": [] } ] }
create an index:
{ "containerlogs-000001": { "settings": { "index": { "ignore_malformed": "true" }, "opendistro": { "index_state_management": { "policy_id": "container-rollover", "rollover_alias": "containerlogs" } }, "refresh_interval": "45s", "number_of_shards": "1", "provided_name": "containerlogs-000001", "creation_date": "1591146104902", "number_of_replicas": "1", "uuid": "G8kDyRntQ_-SZNYXtiIkdw", "version": { "created": "7060199" } } } } }
NOTE: I added the alias at time of creation (doesn’t show here since its the settings of the index)
Check kibana – see the index initializing with the policy…
check the history index:
{ "_index": ".opendistro-ism-managed-index-history-2020.06.02-000005", "_type": "_doc", "_id": "j00kd3IBoht0GWtbMPPz", "_score": 1, "_source": { "managed_index_meta_data": { "index": "containerlogs-000001", "index_uuid": "OiU8IpnOTuqVRBW8vC6QOg", "policy_id": "container-rollover", "policy_seq_no": 10, "policy_primary_term": 4, "state": { "name": "active", "start_time": 1591136760051 }, "retry_info": { "failed": false, "consumed_retries": 0 }, "info": { "message": "Successfully initialized policy: container-rollover" }, "history_timestamp": 1591136760051 } } }
Now that it’s initialized, I start indexing data. The index grows to ~25MB and I refresh the ISM page in kibana… Suddenly my index is attempting to rollover:
Ouptut from explain:
{ "containerlogs-000001": { "index.opendistro.index_state_management.policy_id": "container-rollover", "index": "containerlogs-000001", "index_uuid": "G8kDyRntQ_-SZNYXtiIkdw", "policy_id": "container-rollover", "policy_seq_no": 41, "policy_primary_term": 4, "rolled_over": false, "state": { "name": "active", "start_time": 1591146405278 }, "action": { "name": "rollover", "start_time": 1591146705044, "index": 0, "failed": false, "consumed_retries": 0, "last_retry_time": 0 }, "retry_info": { "failed": false, "consumed_retries": 0 }, "info": { "message": "Attempting to rollover", "conditions": { "min_index_age": { "condition": "1d", "current": "15m", "creationDate": 1591146104902 }, "min_size": { "condition": "1gb", "current": "95.7mb" } } } } }
Pretty sure none of it meets the rollover criteria.
Currently the index is only a few hundred MB, no new index created yet, nothing notable in the logs anywhere. Am I missing something?
Hi @jasonrojas,
Regarding your latest post about it rolling over. Just to clarify did the index actually roll over? The “Attempting to rollover” message is just the generic message when it’s continuously checking conditions. It does not mean it has actually rolled over yet. Once it does rollover you’ll get a message like “Rolled over index” and there will be a log in elasticsearch.log along with an audit log in the history index that contains the conditions that were met when it did rollover.
And for your other points:
The docs for a basic roll-over seem to be lacking a bit around how the index write alias is managed.
Does it have to exist on the index or will ISM add it?
The write alias is something you have to kickstart on the first index you want to have rollover. After that the rollover request will change the is_write_index settings for you when rolling over the index. This is only the case if the alias is pointing to multiple indices. If you have your alias only on your single index then you don’t have to worry about the is_write_index at all.
When an index “Rolls over” for various conditions, does the new index automatically get the alias and the previous index has the alias removed?
If the alias is pointing to a single index then yes, the rollover request will create the new index, add the alias to the new index, and then remove it from the old index.
If the alias is pointing to multiple indices then it will create the new index and use the is_write_index to control which one docs are added to.
Not sure if it was my setup or what but I just tested a daily roll over and I had 5 indices all with the same alias (the one I use to write)
It depends on your original setup. You can read more about rollover here.
Ok, so its been a few martinis but here is what I see.
I changed the policy to 1GB rollover, except instead of specifying “1gb” in the policy, I specified “1000mb” (bug?).
Now, I see that the index has in fact rolled over (I added more indexers to increase the throughput for this test):
That is exactly my setup.
I’m happy to say it is working, albeit the messaging when first checking and the setup instructions drive some confusion. I have a few more tests I have to do for the spec I am working on, and plan to do those either after this drink or in the morning (still undecided)… 
I am more than happy to file bugs if necessary, i am just trying to make sure this as usable as a lot of people hope. Also, if there is a place I can help contribute examples to I will be more than happy to do so.
Hi jason, I try to test your question using a minimal setup
I don’t set "aliases" in index template, only set "opendistro.index_state_management.rollover_alias" to be the one I want to use in rollover;
Then I set "aliases"for my first index. After I add one document, ISM starts to work. It create a new index, set alias for the new index, and remove index from old index, as we would expect.
Final state looks like this:
# test rollover policy
PUT _opendistro/_ism/policies/policy1
{
"policy": {
"description": "test rollover",
"default_state": "rollover",
"states": [
{
"name": "rollover",
"actions": [
{
"rollover": {
"min_doc_count": 1
}
}
],
"transitions": []
}
]
}
}
# create index template
PUT _template/log_template
{
"index_patterns": [
"test_logs-*"
],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 0,
"opendistro.index_state_management.policy_id": "policy1",
"opendistro.index_state_management.rollover_alias": "logs"
}
}
# create test logs index with test rollover policy
PUT test_logs-000001
{
"aliases": {
"logs": {}
}
}
POST logs/_doc
{
"message": "a dummy log"
}
