Question Regarding ISM Policy Compatibility in OpenSearch Version 2.3.0

OpenSearch
, ,
1

Hello OpenSearch Community,
I am currently using OpenSearch version 2.3.0 in my production environment, and I’m interested in implementing ISM (Index State Management) policies, particularly a rollover policy.

However, I am unsure about the compatibility of ISM policies with my current OpenSearch version. Before proceeding with the implementation, I wanted to reach out to the community to inquire whether ISM policies, including rollover policies, are supported in OpenSearch version 2.3.0.

If anyone has experience or insights on this matter, I would greatly appreciate your guidance and advice. It’s crucial for me to ensure that the implementation aligns with my existing setup, and I want to make an informed decision.

2

Yeah, OpenSearch 2.3.0 does support ISM, and also rollover action in ISM policy.

3

Thanks for your reply.
Is ILM is also supported in OpenSearch?

4

ILM, you mean the feature in elasticsearch? No, OpenSearch does not have ILM and other x-pack features of elasticsearch 7.10.2.

5

I’m interested in implementing a data retention policy in OpenSearch, where I can automatically archive data to a new index and delete data from an existing index after a certain time period, for example, 30 days.

I’ve heard about Index Lifecycle Management (ILM) in Elasticsearch, but I’m not sure if the same functionality is available in OpenSearch. Can anyone provide guidance or insights into how I can achieve this in OpenSearch?

6

The functionality of ISM is similar to ILM in Elasticsearch, you can refer to the documentation: Index State Management - OpenSearch documentation

7

I require technical assistance. Is there a method to automatically generate a new index for data that is over 30 days old and subsequently remove that data from the original index? I have experimented with various options within Index State Management (ISM), but it appears that deleting documents from the index is not supported by ISM. Is there a way to automate this task?

8

Delete documents is not supported in ISM, and in my mind ILM in Elasticsearch also doesn’t support delete documents from index. The actions in ISM policy only perform on index level, why not creating monthly index and delete the old index which are older than 30 days? That’s simpler.

If you still want to delete older documents from index, you can setup a scheduler to call delete_by_query API daily:
POST test2/_delete_by_query

{
  "query": {
    "range" : {
        "@timestamp" : {
           "lte" : "now-30d/d"
        }
    }
  }
}