ISM policy applied to data stream keeps initializing and default_state not applied to backing index

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.3.2 / 3.3.0 / Debian 11 / Firefox

Describe the issue:

Hi,

I created the following datastream:

{
  "data_streams" : [
    {
      "name" : "mce-index-w-ds",
      "timestamp_field" : {
        "name" : "@timestamp"
      },
      "indices" : [
        {
          "index_name" : ".ds-mce-index-w-ds-000001",
          "index_uuid" : "OjdN2ex6Q7-jyB1nZBPsJg"
        }
      ],
      "generation" : 1,
      "status" : "GREEN",
      "template" : "mce_logs"
    }
  ]
}

The following policy:

{                                                                                                                                                             [91/1812]
  ".ds-mce-index-w-ds-000001" : {
    "index.plugins.index_state_management.policy_id" : "policy_1",
    "index.opendistro.index_state_management.policy_id" : "policy_1",
    "index" : ".ds-mce-index-w-ds-000001",
    "index_uuid" : "OjdN2ex6Q7-jyB1nZBPsJg",
    "policy_id" : "policy_1",
    "enabled" : true,
    "policy" : {
      "policy_id" : "policy_1",
      "description" : "hot warm delete workflow",
      "last_updated_time" : 1763073417728,
      "schema_version" : 24,
      "error_notification" : {
        "channel" : {
          "id" : "channelexploitantdev"
        },
        "message_template" : {
          "source" : "The index {{ctx.index}} failed during policy execution.",
          "lang" : "mustache"
        }
      },
      "default_state" : "hot",
      "states" : [
        {
          "name" : "hot",
          "actions" : [
            {
              "retry" : {
                "count" : 3,
                "backoff" : "exponential",
                "delay" : "1m"
              },
              "rollover" : {
                "min_doc_count" : 100,
                "copy_alias" : false
              }
            }
          ],
          "transitions" : [
            {
              "state_name" : "warm"
            }
          ]
        },
        {
          "name" : "warm",
          "actions" : [
            {
              "retry" : {
                "count" : 3,
                "backoff" : "exponential",
                "delay" : "1m"
              },
              "replica_count" : {
                "number_of_replicas" : 1
              }
            },
            {
              "retry" : {
                "count" : 3,
                "backoff" : "exponential",
                "delay" : "1m"
              },
              "allocation" : {
                "require" : {
                  "temp" : "warm"
                },
                "include" : { },
                "exclude" : { },
                "wait_for" : false
              }
            }
          ],
          "transitions" : [
            {
              "state_name" : "delete",
              "conditions" : {
                "min_index_age" : "3m"
              }
            }
          ]
        },
        {
          "name" : "delete",
          "actions" : [
            {
              "retry" : {
                "count" : 3,
                "backoff" : "exponential",
                "delay" : "1m"
              },
              "delete" : { }
            }
          ],
          "transitions" : [ ]
        }
      ],
      "ism_template" : [
        {
          "index_patterns" : [
            "mce-index-w*"
          ],
          "priority" : 100,
          "last_updated_time" : 1763073417728
        }
      ]
    }
  },
  "total_managed_indices" : 1
}

All indexes are green:

health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   top_queries-2025.11.12-85602 ukgtr3TwTKuO6XCWtSdyWg   1   2         65           12    898.2kb        299.4kb
green  open   .ds-mce-index-w-ds-000001    OjdN2ex6Q7-jyB1nZBPsJg   2   1          0            0    187.7mb         93.8mb
green  open   .plugins-ml-config           wl0dTHGOTWW-oRF_Wt1rtQ   1   6          1            0     28.4kb            4kb
green  open   .ql-datasources              9Hld-sg6QAaVO6xxX5xJag   1   2          0            0       624b           208b
green  open   top_queries-2025.11.13-85603 XdDiSwbuTkCQwsa5pdhraQ   1   2        904         1563      1.9mb        919.5kb
green  open   top_queries-2025.11.11-85601 1sjWEIpTQUS2Xn6CNBOykg   1   2        185           16      1.2mb        425.9kb
green  open   top_queries-2025.11.10-85600 3PfSpLnHQbCGcSdnjkWlDw   1   2          6            0     72.9kb         24.3kb
green  open   .opendistro_security         8ofRkrvGQoOfM0czuiWukw   1   6          9            0      464kb         66.2kb
green  open   .kibana_1                    _-ySVFglQcuqnVG6r7wFiQ   1   1          3            0     35.3kb         17.6kb

However, the default_state is not applied to the backing index and the policy keeps initializing.

Thanks for any help

Configuration:

Relevant Logs or Screenshots:

image
image1288×327 10.7 KB

image
image744×263 6.06 KB

2

@simelbaz Thank you for the question, an issue has already been raised for this here

As a temporary workaround you can use the following steps:

  • Create new index with mapping from .opendistro-ism-config
  • Reindex .opendistro-ism-config to new index
  • Stop ISM management
  • Delete old index
  • Reindex back into new .opendistro-ism-config
  • Start ISM again
  • ISM working again
  • Rolling restart to remove all locks or blocking processes

I have just tested this and can confirm it remedy the issue. Please note, I manually copied the mappings for temporary index, just to be on a safe side.

Hope this helps

1 Like
3

Hi,

.opendistro-job-scheduler-lock index was missing (probably due to a bad opensearch dashboards install).
After OS Dashboards uninstall/install, the missing index is present and ISM is now OK.

1 Like