Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.9.0
Describe the issue:
I would like create an index policy which deletes documents from index which are older than 30 days without deleting index. The policy should execute everyday to delete documents which are older than 30 days.
Is there any one who has implemented this? I am looking for sample policy.
I think it’s not doable by current ISM policy, because the actions in ISM policy only perform on index level. Why not creating monthly index and delete the old index which are older than 30 days? That’s simpler.
If you still want to delete older documents from index, you can setup a scheduler to call delete_by_query API daily:
POST test2/_delete_by_query
{
"query": {
"range" : {
"@timestamp" : {
"lte" : "now-30d/d"
}
}
}
}
Hello @gaobinlong , Thank you so much for your suggestion. The idea of creating monthly index makes much sense. I would like to go with it. I have set a data prepper to send logs on logs_servers index. How can I create monthly index for that? Thanks.
You can achieve that by specifying the index parameter to a date-time pattern, like this:
index: my-index-name-%{yyyy.MM}
, see more here: https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/README.md
Hello @gaobinlong , Thank you so much for the suggestion. I tried specifying index parameter as my-index-name-%{yyyy.MM.dd.HH} in data prepper and it’s showing while trying to create index in OpenSearch but I need to create the index manually specifying the timestamp field explicitly as below:
PUT my-index-name-2023.09.06.06
{
"mappings": {
"properties": {
"log_time": {
"type": "date",
"format": "[dd/MMM/yyyy:HH:mm:ss.SSS]"
},
"bytes": {
"type": "integer"
}
}
}
}
My index policy is like below:
{
"policy_id": "index_rotation_everhour",
"description": "Hot/Warm/Delete example",
"last_updated_time": 1693976276550,
"schema_version": 18,
"error_notification": {
"channel": {
"id": "SVGfG4oBbMQGkbGGjItQ"
},
"message_template": {
"source": "Index {{ctx.index}} failed",
"lang": "mustache"
}
},
"default_state": "hot",
"states": [
{
"name": "hot",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"rollover": {
"min_index_age": "2h",
"min_primary_shard_size": "5gb"
}
}
],
"transitions": [
{
"state_name": "warm",
"conditions": {
"min_rollover_age": "1h"
}
}
]
},
{
"name": "warm",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"replica_count": {
"number_of_replicas": 0
}
}
],
"transitions": [
{
"state_name": "delete",
"conditions": {
"min_rollover_age": "3h"
}
}
]
},
{
"name": "delete",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"notification": {
"channel": {
"id": "SVGfG4oBbMQGkbGGjItQ"
},
"message_template": {
"source": "Index: {{ctx.index}} Deleted",
"lang": "mustache"
}
}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"my-index-name-*"
],
"priority": 100,
"last_updated_time": 1693934657602
}
]
}
For the first time, I created index my-index-name-2023.09.06.06 manually and I was expecting the next index my-index-name-2023.09.06.07 will be created automatically because of index policy but it’s not happening. While trying to create manually it shows my-index-name-2023.09.06.07 was available in index pattern but it does not have timestamp (log_time in my case) and it looks like it’s been already created without timestamp as it’s not asking me to select timestamp but if it was already created it should be available on the discover menu but it’s not there. I will be very grateful if you have any suggestion on this.
Thanks for your help.
The rollover action in ISM cannot rollover my-index-name-2023.09.06.06 to my-index-name-2023.09.06.07, the index name must end with - and a number, like this: logs-000001, then the rollover action can work. If you specify the index parameter to my-index-name-%{yyyy.MM.dd.HH} in data prepper, that means data prepper will create index hourly, and you do not need to add rollover action to your ISM policy.
And for the timestamp field in your index, you can define a index template(not ism template) to match your indices, every new index will have same mappings and settings.
Hello @gaobinlong , Thank you so much for your help and I really appreciate it. I was able to fix the issue for the log rotation. Now, I have a final problem, I see that index are being created with correct mappings but I need to go to Dashboard Management > Index Patterns and create index from Create index pattern menu selecting the timestamp for the index to make the index available in discover menu in the Opensearch dashboard. Is there any way, we can make it automatically? Once the index is created, it should be available in discover menu automatically. Thanks.
Thank
I think the index pattern needs to be created only once, the new created index which matches the index pattern can be searched in Discover immediately, but if your purpose is to create an index pattern in OSD once there is a new index, you can call the create saved object API of OSD to create an index pattern, like this:
curl -X POST localhost:5601/api/saved_objects/index-pattern/my-pattern -H 'osd-xsrf: true' -H 'Content-Type: application/json' -d '
{
"attributes": {
"title": "my-pattern-*"
}
}'
.