Is it possible to delete of open search data streams using ISM policy?

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
opensearch version : 2.12

Describe the issue:
I have created a data stream in opensearch using API command.
now I have a writing index in the data stream where the logs are getting ingested, let’s say the rollover condition not reached for many days, and also everyday new data steams would getting created resulting that many writing index for those data streams, in this scenario no index would be able to get deleted by ISM policy defined, so that in my system many data streams and their writing index would get piled up eventually.

how do I clean up the data streams itself in this case automatically instead of using API command to delete the data streams.

Configuration:

Relevant Logs or Screenshots:

1 Like
2

hi, any update on this query?
can someone help to understand the behaviour?

3

@subha_1, If I got it right, you could use something like:

{
  "policy": {
    "description": "Delete indices after X days",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "30d" #number of days to keep index
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": ["data-stream-prefix-*"],
      "priority": 100
    }
  }
}

best,
mj

4

hi @Mantas , Thanks for your reply.
the ISM policy which you have mentioned is just deleting the backing indices in the data stream as per the condition given in policy, it’s not deleting the complete data stream itself.

my requirement is to delete the complete data stream using ISM policy, and wanted to check if it would support?