Error while rollover action. How to create rollover alias?

Plugins Index Management
1

Hi,

I have integrated metricbeat and I created a template before sending index to the system. I am getting indices like metricbeat-7.9.1-2020.11.26. I have attached a ISM Policy, so that my expectation is, to create a new index metricbeat-7.9.1-2020.11.26-00001 when the current index cross 100mb size. Metricbeat indices will get created daily, like metricbeat-7.9.1-2020.11.26, metricbeat-7.9.1-2020.11.27, metricbeat-7.9.1-2020.11.28 etc. SO, I want to split up every daily index, based on size. I created the policy like below;

{
    "policy_id": "metricbeat",
    "description": "metricbeat test policy",
    "last_updated_time": 1606385808719,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "rollover": {
                        "min_size": "100mb"
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "5d"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ]
}

and I received an error while trying to attempt rollover, like the alias does not exist. Then, I updated thhe metricbeat template by adding the template setting like below;

{
    "metricbeat-7.9.1" : {
        "aliases": {
            "metricbeat": {}
        }
    }
}

Now, I am getting error on ISM retry;

{
    "cause": "rollover target [metricbeat] does not exist",
    "message": "Failed to rollover index [index=metricbeat-7.9.1-2020.11.26]"
}

Why is this happening? How can I fix this?

Thanks in advance.

2

Your rollover target (you called it ‘metricbeat’) is not associated with any index. In fact, it’s just an alias and you must “kickstart” it by assigning it to an index.
Run API
POST /_aliases
{
“actions”: [
{
“add”: {
“index”: “<index name to which the alias should be pointed to”,
“alias”: “your alias name (which is also your rollover target)”,
“is_write_index”: true
}
}
]
}

Please also note that your index name pattern must end with -d+ (see Rollover API | Elasticsearch Guide [master] | Elastic)
And I did not find a way to make it automatically (by setting the respective definitions in the beat’s configuration file), so once I reset indexes for some reason (no metricbeat indexes in ES cluster at all), this must be done manually again.
Tell me if it works for you because for me it rolls over and creates a new index but keeps on writing to the old index (e.g. if I had filebeat-000001, it creates a new filebeat-000002 but no data is in the new index and all keeps on going to the old filebeat-000001. Once filebeat-000001 is deleted according to its policy, it gets recreated and being written to again)

3

Hi @igorid70

For me also, new index got created (rolled over) but still, data is getting published to the old index and the new index has 0 documents. Why is this happening?

4

I am breaking my head for more than one week and still no answer. I migrate a “regular” Elastic stack to Open Distro due to license issue (Apache 2.0 vs. Elastic license) and I must admit that it is a complete disappointment. For example, the part of Index management took for me around 20 minutes to learn and configure in Elastic stack with its ILM (Index Lifecycle maintenance). How people use Open distro if you can’t find any documentation beside the general explanations? Isn’t the basic feature of rolling indexes over implies also writing to the new index which was just rolled over? Someone?

5

I agree. ILM was somewhat easier to learn than ISM I believe, also, the documentation of OD is “hard to digest”. What is missing here is step-by-step guides. For example, its not so hard for them to publish a guide / blog to achieve ISM in Filebeat / Metricbeat, which might be helpful for so many people.

I was just wondering, is it possible to create daily indices based on template and alias. FOr example, if I have an alias myalias pointing to myindex-000001, if I publish messages to myalias, finally the data will be published to myindex-000001 for sure, and if ISM is successful, it will create myindex-000002, myindex-000003 and so on. What I am trying to achieve is, is it possible to append datestamp before the index number and after index name, like, myindex-2020.12.01-00001, myindex-2020.12.01-00002, myindex-2020.12.02-00001, myindex-2020.12.03-00001, etc… Daily indices will get created, along with numbering. Is this possible?

1 Like
6

Yes, it is possible. You need to specify something like this in your metricbeat.yml
output.elasticsearch.index: "index: “metricbeat-%{+yyyy.MM.dd}-000001”
You can play with the name as you wish (based on ES restrictions about the special characters, etc), but if you want the rollover feature, your index name must end with a digit (and during rollover, the last digit will be increased by one and the number will become 6 digits). But of course as per our common issue, once rolled over, the new index will stay “empty” meaning it will keep on writing to the old one. I will try to post a new question about the proper way (if there is such a way) to setup the automatic rollover (when the write index is the one we just rolled over to)

7

Hello @igorid70 ,

Were you able to make any positive progress on this issue? I seem to be having similar issues so was curious before I reached out to support. We basically want to rollover the indexes once per day OR increment up multiple times per day if the index is over 10GB which-ever comes first.

For Example:
<index_name> — <index_size>
myindex-2020.12.01-000001 — 10GB
myindex-2020.12.01-000002 — 8GB
myindex-2020.12.02-000003 — 3GB
myindex-2020.12.03-000004 — 10GB
myindex-2020.12.03-000005 — 10GB
myindex-2020.12.03-000006 — 3GB
myindex-2020.12.04-000007 — 5GB

This is mostly how ILM would setup the indexes and how we want to replicate it, other suggestions are welcome. We mainly are looking for the best option to store 30 days of data, with a somewhat fixed Index size for performance.

Thanks!
-Chris

8

Hi @ChrisCarter
Unfortunately, nobody answered in this forum and also in official elastic forum where I published the same question. I really don’t understand how people implement Opendistro without such a basic thing. What do they do with the indices which keep on getting accumulated? For me it was a definite ‘no go’ for Opendistro but our legal department did not allow to use Elasticsearch in its basic subscription due to the limitation in Elastic license (BTW, I am very pleased with all the rest in Opendistro).
Anyway, I ended up by writing a Perl script using the official Elasticsearch Perl client which reads the configuration file where I define the cleanup rules (based on size or age of an index/index pattern) and it deletes the indices if the conditions are met. For the indices themselves I left the default configuration, so beats produce daily indices, i.e. filebeat-7.9.1-2020.12.18, metricbeat-7.9.1-2020.12.18, heartbeat-7.9.1-2020.12.18 which of course can’t be rotated because they don’t match the pattern required for rotation.

9

Hi !

I don’t use metricbeat but logstash which works the same way.
You have to use aliases in metricbeat instead of index names. An alias could be as simple as metricbeat.
Manually bootstrap an index using index API that uses that alias and set is_write_alias to true. That first index’s name must end with some digits like -00001.
Then, from kibana UI or using ISM API, define a policy that will do rollover based on size and date, and deletion after some time.
The alias will be automatically applied to the new index created by your policy.
See Redirecting…

Your logs/metrics collector agents shouldn’t care about indexes…

Cheers

10

Hi @piequi

Thanks for trying to assist.
Assigning alias name to output.elasticsearch.index was one of the first things I tried and I was quite surprised when ES created an index with the same name as the alias defined in output.elasticsearch.index. Of course I did not get what I wanted as such an index can’t be rolled over at all (not matching -d+ pattern at the end of the name).

11

Hi @igorid70

If you want to achieve rolling a simple alias over an index made of a timestamp and an incrementing ID like my-index-{now/M{yyyy.MM}}-000001 you’ll have to do it in 2 steps. There is no way you can specify such a pattern in ISM in rollover operation.

  1. You have to create an external scheduled job to call rollover API specifying with some date math to create a new index with the updated timestamp when needed. (This doc is valid for OpenDistro as well)
  2. You have to create an ISM policy with a pattern matching all your timestamped indexes to do the rollover based on the index size.

I didn’t deployed that in production as I actually ended up dumping the timestamp. I use only one ISM simple policy with an incrementing ID. This is totally fine.

Cheers

12

Hi
Thanks for the clarification. This basically means that contrary to XPACK ILM where the whole process is absolutely transparent and achieved by pure configuration definitions in beats (and I guess in logstash as well, I just don’t use it), Opendistro requires an “external” process being involved - this is what I understand you meant saying “You have to create an external scheduled job”, e.g. crontab activating rollover API or something like this, which is of course not you want to have in your system. For my use case I will keep on using my own implementation of index management and index cleanup. If I can’t avoid having a process which runs in addition to my Elastic stack itself, I prefer to stay with what I specifically tailored for indices. But it really makes sense if the whole process of index policy is based only on configuration without activating any additional processes (like rollover API). Maybe Opendistro will have it in future releases.