Need some help with ISM Policy

OpenSearch
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.13

Describe the issue:

We currently have new indices created everyday, with the date suffix, and there is a migration policy based on time. How to add another component to this policy, which is rollover, based on sizes

How can I change index template and ISM Policy to handle this feature?

creates an index with a size limit of 70GB. Once the index reaches the limit, a new index should be created with the same date followed by an incrementing suffix (e.g., http-2024-09-16, http-2024-09-16-1, http-2024-09-16-2). This should continue for each new index once the size limit is reached."

Configuration:
ISM Policy:

{
“policy_id”: “http-indices-policy”,
“description”: “Move http indices to warm and Delete after 180 days from cold.”,
“last_updated_time”: 1706223240662,
“schema_version”: 19,
“error_notification”: null,
“default_state”: “http-indices”,
“states”: [
{
“name”: “http-indices”,
“actions”: ,
“transitions”: [
{
“state_name”: “warm-http-indices”,
“conditions”: {
“min_index_age”: “3d”
}
}
]
},
{
“name”: “warm-http-indices”,
“actions”: [
{
“timeout”: “24h”,
“retry”: {
“count”: 5,
“backoff”: “exponential”,
“delay”: “1h”
},
“warm_migration”: {}
}
],
“transitions”: [
{
“state_name”: “cold-http-indices”,
“conditions”: {
“min_index_age”: “23d”
}
}
]
},
{
“name”: “cold-http-indices”,
“actions”: [
{
“timeout”: “24h”,
“retry”: {
“count”: 5,
“backoff”: “exponential”,
“delay”: “1h”
},
“cold_migration”: {
“start_time”: null,
“end_time”: null,
“timestamp_field”: “@timestamp”,
“ignore”: “none”
}
}
],
“transitions”: [
{
“state_name”: “delete-http-indices”,
“conditions”: {
“min_index_age”: “206d”
}
}
]
},
{
“name”: “delete-http-indices”,
“actions”: [
{
“timeout”: “24h”,
“retry”: {
“count”: 5,
“backoff”: “exponential”,
“delay”: “1h”
},
“cold_delete”: {}
}
],
“transitions”:
}
],
“ism_template”: [
{
“index_patterns”: [
“http-",
"fluentbit-osi-http-”
],
“priority”: 0,
“last_updated_time”: 1648182660976
}
]
}

IndexTemplate Settings:
{
“fluentbit-osi-http-2024.09.16”: {
“settings”: {
“index”: {
“replication”: {
“type”: “DOCUMENT”
},
“codec”: “best_compression”,
“mapping”: {
“ignore_malformed”: “true”
},
“refresh_interval”: “45s”,
“number_of_shards”: “45”,
“provided_name”: “fluentbit-osi-http-2024.09.16”,
“creation_date”: “1726444800546”,
“number_of_replicas”: “1”,
“uuid”: “17zHCD2JRfKnFRh6iZt24A”,
“version”: {
“created”: “136347827”
}
}
}
}
}

Relevant Logs or Screenshots:

2

Hi @rvn,

If I got it correctly, you just add a rollover action to your ISM policy:

i.e.:

"actions": [
  {
    "rollover": {
      "min_size": "70gb"
    }
  }
]

it would look something like:

{
  "name": "http-indices",
  "actions": [
    {
      "rollover": {
        "min_size": "70gb"
      }
    }
  ],
  "transitions": [
    {
      "state_name": "warm-http-indices",
      "conditions": {
        "min_index_age": "3d"
      }
    }
  ]
}

See more here: Policies - OpenSearch Documentation

Best,
mj