Hi Team,
I need your help or any experince on that . I want to configure ISM. is it must to use data stream? I’m getting logs via filebeat and log stash. Logstash create daily base index name like “servername-date XX-YY-ZZZ” like apigw-02022023. Can I configure ISM for these index names?
I tried to create ISM policiy with min_age and min_index size. When I run it, policy remove all the index without looking index age or index size.
Thanks in advance.
Regards,
Emre
Hello @a.emrekaraman,
You are not required to use data streams. I have a similar example configuration where I am creating a new index for every day of data for auditbeat:
I also have an ISM Policy for automatically deleting anything older than a certain amount of days. This is the policy I use:
{
"id": "auditbeat",
"seqNo": 447169,
"primaryTerm": 15,
"policy": {
"policy_id": "auditbeat",
"description": "Delete auditbeat logs after configured period of time.",
"last_updated_time": 1676465660005,
"schema_version": 17,
"error_notification": null,
"default_state": "hot_state",
"states": [
{
"name": "hot_state",
"actions": [],
"transitions": [
{
"state_name": "delete_state",
"conditions": {
"min_index_age": "4d"
}
}
]
},
{
"name": "delete_state",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"auditbeat-*"
],
"priority": 1,
"last_updated_time": 1675959511978
}
]
}
}
“hot_state” is the state where I keep the index, “delete_state” when I want to delete them. This works for me without issue.
Could you maybe share your configuration with us?
Hello,
Thank you very much for response. I stuck at this step 
My index names like;
green open filebeat-tmktstweb01-7.12.1-2023.01.28 gOymuBF3QbiUJ5ObsAuHEQ 1 1 44233 0 21.4mb 10.3mb
green open filebeat-tmktstweb01-7.12.1-2023.02.18 ozbv6wtFQ9O869CJ38HJMA 1 1 83699 0 60.2mb 30mb
green open filebeat-tmktstweb01-7.12.1-2023.01.29 58lWjvTERbmDQkUyRS7UQQ 1 1 71007 0 34.5mb 17.1mb
green open filebeat-tmktstweb01-7.12.1-2023.02.19 r0su9qf7RZmWrml3mqA70g 1 1 55817 0 38.4mb 19.3mb
green open filebeat-tmktstweb01-7.12.1-2023.01.30 FN1rjv6gTRezzZIlgfnr_A 1 1 72348 0 36.9mb 18.4mb
green open filebeat-tmktstweb01-7.12.1-2023.02.20 kOVzzd7XSHSTgAF3b09DfA 1 1 18557 0 9.2mb 4.5mb
green open filebeat-tmktstweb01-7.12.1-2023.02.05 tSadfkdESDOOFilSQOKbMw 1 1 82471 0 36.7mb 18.4mb
green open filebeat-tmktstweb01-7.12.1-2023.02.06 2sHPoyCdQuqH55Kf-cCNRA 1 1 132860 0 63.1mb 31.5mb
I create policy. this first move indexes close state with min_age 3 days after that remove indeces which was closed but when I ran it, it close all indeces and remove without looking min_age.
{
“id”: “cold”,
“seqNo”: 51496,
“primaryTerm”: 1,
“policy”: {
“policy_id”: “cold”,
“description”: “A sample description of the policy”,
“last_updated_time”: 1676882504554,
“schema_version”: 17,
“error_notification”: null,
“default_state”: “close”,
“states”: [
{
“name”: “close”,
“actions”: [
{
“retry”: {
“count”: 3,
“backoff”: “exponential”,
“delay”: “1m”
},
“close”: {}
}
],
“transitions”: [
{
“state_name”: “delete”,
“conditions”: {
“min_index_age”: “3d”
}
}
]
},
{
“name”: “delete”,
“actions”: [
{
“retry”: {
“count”: 3,
“backoff”: “exponential”,
“delay”: “1m”
},
“open”: {}
}
],
“transitions”:
}
],
“ism_template”: [
{
“index_patterns”: [
“filebeat-tmk*”
],
“priority”: 100,
“last_updated_time”: 1676454534009
}
]
}
}
For example, I created same policy for min_age 10 day. it just ran for few indeces. Normally, there are filebeat-tmk* 30 days indeces. but policy ran for last 6 day indeces.
Hello again,
your configuration would instantly close all indices matching the pattern, and after 3 days open them again.
I imported your config to showcase this in the visual editor:
We can see, that every index woud start as “closed” and transition to the “delete” state after 3 days, which would open them again.
What exactly is your goal, what do you want to achieve with your ISM? Do you want do delete every filebeat-tmk* index after 3 days?
Furthermore you need to wait for some time for the ISM to apply all policies to all existing indices.
BR,
Andreas
Hi,
What my goal is;
1- closed every index filebeat-tmk* after 3 days. I mean today 20 FEB. I want to close all indeces before 17 FEB. for example indeces belong to 16 FEB,15 FEB,14 FEB … should be closed.
2- All closed indeces should be deleted whenever indeces are closed. no need to wait to close them.
Sorry. Let me share current policy details;
{
“id”: “cold”,
“seqNo”: 52031,
“primaryTerm”: 1,
“policy”: {
“policy_id”: “cold”,
“description”: “A sample description of the policy”,
“last_updated_time”: 1676888474312,
“schema_version”: 17,
“error_notification”: null,
“default_state”: “close”,
“states”: [
{
“name”: “close”,
“actions”: [
{
“retry”: {
“count”: 3,
“backoff”: “exponential”,
“delay”: “1m”
},
“close”: {}
}
],
“transitions”: [
{
“state_name”: “delete”,
“conditions”: {
“min_index_age”: “3d”
}
}
]
},
{
“name”: “delete”,
“actions”: [
{
“retry”: {
“count”: 3,
“backoff”: “exponential”,
“delay”: “1m”
},
“open”: {}
}
],
“transitions”:
}
],
“ism_template”: [
{
“index_patterns”: [
“filebeat-tmk*”
],
“priority”: 100,
“last_updated_time”: 1676454534009
}
]
}
}
Thnaks,
Regards,
Emre
I dont think I fully understand your goal.
So I currently think it should work like this:
Current date: 20.02.2023
Index-20.02.2023: keep
Index-19.02.2023: keep
Index-18.02.2023: keep
Index-17.02.2023: keep
Index-16.02.2023: close & delete
correct?
So indices from the 16.02.2023 and older will be closed AND deleted.
If so the following configuration should work for you:
{
"id": "cold",
"seqNo": 680598,
"primaryTerm": 15,
"policy": {
"policy_id": "cold",
"description": "A sample description of the policy",
"last_updated_time": 1676903873855,
"schema_version": 17,
"error_notification": null,
"default_state": "keep",
"states": [
{
"name": "keep",
"actions": [],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_index_age": "3d"
}
}
]
},
{
"name": "delete",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"close": {}
},
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"filebeat-tmk*"
],
"priority": 100,
"last_updated_time": 1676454534009
}
]
}
}
This would transition indices after the configured days to the delete state, where they would be closed and removed.
Hi,
Yes, you’re correct. I tested policy you shared and it worked. Thank you very much. I’m appreciated 
I just have few questions;
1- I dont understand logic of transitions. that’s why my first policy failed:(
2- I ran the policy which you shared and wait since yesterday but it just worked for few index not for all of them. There are index for 30 days but policy deleted 3 of them. How long should I work for policy to applt all of them? is there a default vaule for timing?
3-Also policy remove 3 day and older index. but policy always for for 3 day and newer index forever. will it stop work on newer indeces?
Great to hear
ok as to your further questions:
1 - When creating a policy for an index it always has to be in a certain state, this would be the “default_state”: “keep” line in my configuration. Now that my index is in the “keep” state it looks at what actions it needs to do. My array here is empty so no actions will be executed. Then it looks at the transitions when it transitions to the next state, in my case this happens when the “min_index_age” is 3d, then it would transition to the “delete” state.
The same thing happens again - we look at the actions, which contain “close” and then “delete”. Since an index only gets to the “delete” state after 3d it would only delete them after this period. Here is a simple sketch to visualize this:
2 - This question is a little tough for me. I think I cant give you an answer for this, sorry. I just know it took a long time in my cluster as well, but works great with any new indices.
3 - No, it will not stop working on newer indices. Every day the oldest Index will be deleted and the new index will be added to the “keep” state. So you will always have a total of 3 indices.
BR,
Andreas
