Using ISM with indices that contain date in the name

General Feedback
1

Good day all,

My goal is to keep the last 7 days of data, each in an index whose name contains the date. After 5 days, I’d like the index automatically removed.

This is similar to Policies - OpenSearch documentation, except with the auto-increment of a date in the index instead of an autoincrement of a number.

First, I create an ISM policy:

PUT _plugins/_ism/policies/daily_foobar_rollover_policy
{
    "policy": {
        "policy_id": "daily_foobar_rollover_policy",
        "description": "hot warm delete workflow",
        "schema_version": 15,
        "error_notification": null,
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "rollover": {
                            "min_index_age": "1d"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "warm",
                        "conditions": {
                            "min_index_age": "1d"
                        }
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "replica_count": {
                            "number_of_replicas": 2
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "7d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "foobar*"
                ],
                "priority": 100
            }
        ]
    }
}

Then, I make a template that references the policy and provides the name of the rollover alias

PUT _index_template/daily_tags
{
  "index_patterns": [
    "foobar*"
  ],
  "template": {
    "aliases": {
      "foobar": {}
    },
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1,
      "index.opendistro.index_state_management.policy_id": "daily_foobar_rollover_policy",
      "opendistro.index_state_management.rollover_alias": "foobar"
    },
    "mappings": {
      "properties": {
        "timestamp": {
          "format": "date_time",
          "type": "date"
        },
        "namespace": {
          "type": "text"
        },
        "tagName": {
          "type": "text"
        },
        "tagValue": {
          "type": "text"
        }
      }
    }
  }
}

Finally, I create a ‘bootstrap index’ , using the date-time syntax I found at API conventions | Elasticsearch Guide [master] | Elastic .

I’d like foobar-yyyy-MM-dd.
I thought this could be achieved with the URL-encoded version of
" PUT /<foobar-{now/d{yyyy-MM-dd}}>"

The following does successfully create the expected index (for this example, foobar-2022-08-24)

PUT /%3Cfoobar-%7Bnow%2Fd%7Byyyy-MM-dd%7D%7D%3E
{
  "aliases": {
    "foobar": {
      "is_write_index": true
    }
  }
}

But a day later, the rollover fails with this message:

{
    "cause": "For input string: \"dd}}\"",
    "message": "Failed to rollover index [index=foobar-2022-08-24]"
}

Any advice much appreciated!

2

FWIW, a simpler index pattern also fails at rollover. I suspect that opensearch rollover simply doesn’t support indices with dates in the name.

PUT /%3Cfoobar-%7Bnow%2Fd%7D%3E
{
  "aliases": {
    "tagged-emails": {
      "is_write_index": true
    }
  }
}

successfully creates the index foobar-2022.08.25

However, the subsequent rollover fails with a clearer, but disappointing error message:

{
    "cause": "index name [<foobar-{now/d}>] does not match pattern '^.*-\\d+$'",
    "message": "Failed to rollover index [index=foobar-2022.08.25]"
}