Hi
I am trying to configure Beats to have an index policy which rotates the active index (the one which is written to at the moment) based on size/age criteria and later deletes them based also on size/age. I use ODFE 1.10.1 and OSS Beats 7.9.1
The problem is that no matter what I tried, Beats starts writing to the first index (e.g. metricbeat-000001) which is created after I start the beat for the first time and keeps on writing to it. Another words, it applies rollover operation and successfully creates metricbeat-000002 index according to the policy, applies is_write_index=true to metricbeat-000002 and changes is_write_index=false to the original metricbeat-000001. However, metricbeat-000002 remains empty and all the data keeps on going to metricbeat-000001. All the indices continue with their lifecycle as per the policy (new mectribeat-000003, metrcbeat-000004, etc. get created and deleted as expected but they all are empty, i.e. nothing writes to them). When the first metrcibeat-000001 gets deleted according to policy, it then gets recreated and it all starts over - the data is written to this newly recreated metricbeat-000001.
So, it looks that I can’t make it write to index alias rather than to index. I also tried to define alias in output.elasticsearch.index but I was quite surprised that it created index with the name I gave to alias (so I had alias and index with the same name) but still it wrote to the index (which had the same name like the alias) with the same behavior.
My beat (here is metricbeat) configuration snippet
output.elasticsearch
index: “metricbeat-000001” #also tried with my indices alias name
setup.template.settings:
index.opendistro.index_state_management.rollover_alias: “alias_metricbeat”
index.opendistro.index_state_management.policy_id: “metricbeat_index_policy”
opendistro.index_state_management.history.rollover_retention_period: “7d”
setup.template.type: “index”
setup.ilm.enabled: false
setup.ilm.check_exists: false
my policy (the short times are for the test)
{
“policy”: {
“policy_id”: “metricbeat_index_policy”,
“description”: “A metricbeat indexes workflow.”,
“last_updated_time”: 1606754247231,
“schema_version”: 1,
“error_notification”: null,
“default_state”: “hot”,
“states”: [
{
“name”: “hot”,
“actions”: [
{
“rollover”: {
“min_index_age”: “10m”
}
}
],
“transitions”: [
{
“state_name”: “delete”,
“conditions”: {
“min_index_age”: “2h”
}
}
]
},
{
“name”: “delete”,
“actions”: [
{
“delete”: {}
}
],
“transitions”:
}
]
}
}
Any help, suggestions or ideas are highly appreciated.