Hi team!
After reading through the docs attempted the rollover using ISM policy as follows. But it seems the index does get rolled over from test-000001 to test-000002 but keeps writing to test-000001 and the index keeps growing. Here are the steps I followed:
1] Created a policy:
{
"policy": {
"policy_id": "rollover_policy",
"description": "Rollover test",
"last_updated_time": 1597680827746,
"schema_version": 1,
"error_notification": null,
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"retry": {
"count": 10,
"backoff": "exponential",
"delay": "1h"
},
"rollover": {
"min_size": "300gb"
}
}
],
"transitions": [
{
"state_name": "warm",
"conditions": {
"min_index_age": "5d"
}
}
]
},
{
"name": "warm",
"actions": [
{
"retry": {
"count": 5,
"backoff": "exponential",
"delay": "1h"
},
"warm_migration": {}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "10d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"delete": {}
}
],
"transitions": []
}
]
}
}
2] Created an index and attached the policy to it:
PUT test-000001
{
"settings": {
"opendistro.index_state_management.policy_id": "rollover_policy",
"opendistro.index_state_management.rollover_alias": "demo"
},
"aliases": {
"demo": {
"is_write_index": true
}
}
}
3] Registered a template (for upcoming indices to have the same policy):
PUT _template/template_1
{
"index_patterns": [
"test-*"
],
"settings": {
"index": {
"opendistro": {
"index_state_management": {
"policy_id": "rollover_policy",
"rollover_alias": "demo"
}
}
}
}
}
Tried a couple of variations like from the Index Management (IM) tab in Kibana, attaching the policy and the rollover_alias to the index: test-000001 as mentioned here But it seems I am missing some key step which is not making the rollover_alias point to the rolled over index
Here is the logstash conf:
amazon_es {
hosts => ["http://myelasticsearch"]
index => "test-000001"
}
Observation:
Docs are successfully ingested to the index: test-000001 and after the primary reaches ~300gb, a new index gets created test-000002 however, docs continue to be ingested in test-000001 (Apart from this, the migration to warm and then delete all works smooth)
Any help would be appreciated.