Index rollover in opensearch using ISM with indexes using datemath

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.13

Describe the issue:
I am working on to create index rollover for the indexes which grows in size. I am able to get the correct policy to do so. I am stuck in applying the policy to indexes and setting rollover alias. so from the docs i found rollover onlyworks if the index names matches the pattern - ^.*-\d+$. for example it should have something like logs-00001 for initial index and when rollover is done the index is logs-00002 and the alias points to the current index. i am pushing indexes and data to it from logstash pipeline like this -
opensearch {
index => “logs-%{[@metadata][target_index]}-%{+YYYY.MM}”
hosts => [“${ELASTICSEARCH_HOST}:443”]
ssl => true
auth_type => {
type => ‘aws_iam’
region => “${AWS_REGION}”
}
}
i already have indexes created something like - logs-2024.09.10
which is why i am not able to perform the rollover actions
How can i solve this i mean how can i do the rollover with indexes like this.
so when i get an index pushed on any date i want it should have a initial index like - logs-2024.09.10-000001
and this will be the initial index on each index.
Also to ensure when the rollover is done the data is written to the active idex and not the old index

Configuration:

Relevant Logs or Screenshots:

2

Hi there!

There’s 3 key steps that I would suggest;

  1. Configure an ISM polisy
  2. Configure an Index template (with a defined rollover_alias)
  3. Create the initial write index

I’ve done this all in dev tools, here are the relevant commands;

Configure ISM policy

PUT _plugins/_ism/policies/my_ism_policy
{
  "policy": {
    "description": "Rollover after 5 min and transition to inactive then delete after 30min",
    "default_state": "active",
    "states": [
      {
        "name": "active",
        "actions": [
          {
            "rollover": {
              "min_index_age": "5m"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "inactive",
            "conditions": {
              "min_index_age": "5m"
            }
          }
        ]
      },
      {
        "name": "inactive",
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "30m"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ]
      }
    ],
    "ism_template": {
      "index_patterns": [
        "rollover-*"
      ],
      "priority": 100
    }
  }
}

Configure index template

PUT _index_template/rollover_template
{
  "index_patterns": [
    "rollover-*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "plugins.index_state_management.rollover_alias": "rollover_alias"
    }
  }
}

Create initial write index

PUT %3Crollover-%7Bnow%2Fd%7D-000001%3E
{
"aliases": {
 "rollover_alias": {
   "is_write_index": true
        }
    }
}

From there, your logstash config index setting should point to your rollover alias (in the example above, is being defined as “rollover_alias”).

Hope this helps,
Jake

3

Is there any way to automate the initial index creation as for each day i need to do the put req for each index daily. I don’t want to use any custom script which will do the put operation

4

I’d encourage you to try this and experiment with the results. You are required to define an initial index as a write index, but you don’t need to make the PUT request again after the initial creation.

5

will the above work with daily indices created by -
opensearch {
index => “%{[@metadata][target_index]}-%{+YYYY.MM.dd}”
hosts => [“${ELASTICSEARCH_HOST}:443”]
ssl => true
auth_type => {
type => ‘aws_iam’
region => “${AWS_REGION}”
}
}
since indexes are created daily ??