Retention Policy at Index level

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): v 2.11.0

I am managing an active index from fluent bit, I want to set up a policy where the index should always contain the last 30 days of logs in active index. Each day, the oldest logs from the active index need to be removed and moved to an inactive index, which then starts storing the removed data. This rotation should occur daily to ensure the active index only stores the most recent 30 days of data without affecting its performance or moving it to warm state.

2

Hey @spoorthi

What have you tried so far?

Use the RollOver

{
  "rollover": {
    "min_index_age": "30d"
  }
}

transition

"transitions": [
  {
    "state_name": "warm",
    "conditions": {
      "min_index_age": "30d"
    }
  }
]

Perhaps something like this example here

1 Like
3

I did try rollover but I want the index to keep receiving data, I cant close the index as rollover makes the index go into warm state, also I wanted to create daily indexes (since ISM works only at index level) and keep it in a warm state for 35 days , making sure I always have monthly logs for querying,

4

Hey

The link I posted show you something similar to what you want.

5

Thank you

1 Like
7

Hey @spoorthi

the index name does not follow the rules of the rollover criteria.

This is about three years old but looks like your issue