SM policy intermittently not attaching to newly created indices in OpenSearch

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch version: 2.19.0

OpenSearch Dashboard version: 2.17.0

Describe the issue:

We have defined an ISM policy and configured it through index templates.

We observed that for a few indices, the ISM policy is not getting attached, even though:

  • The ISM policy existed before index creation

  • The index template correctly references the policy

  • Most indices get the policy attached successfully

However, a small number of indices are created without any ISM policy attached.

This behavior appears to be random and inconsistent. Same ISM policy, same index pattern, same source of index creation but no identifiable pattern based on time or index name. These indices remain unmanaged unless the policy is manually attached later.

We suspect this could be related to a race condition or a bug in ISM’s background policy attachment mechanism.

Questions:

  • Is this a known issue in OpenSearch ISM?

  • Are there known race conditions or scheduler limitations that could cause this?

  • Are there recommended safeguards or workarounds?

Configuration:

One of the use cases is here:

opensearch-configs:
  ismPolicies:
    - name: proxy-sftp-access-ism-policy
      description: sftp proxy access index management policy
      defaultState: init
      ismTemplate:
        indexPatterns:
          - proxy-sftp_access*
        priority: 100
      states:
        - name: init
          actions: []
          transitions:
            - stateName: delete
              conditions:
                minIndexAge: 120d
        - name: delete
          actions:
            - delete: {}

Relevant Logs or Screenshots:

image-2026-01-08-16-25-56-408
image-2026-01-08-16-25-56-4081472×571 75.7 KB

2

@ Anthony, I see you gave a comment previously to check these values

GET /.opendistro-ism-config/_doc/?pretty and I did run this is the output
{
“_index”: “.opendistro-ism-config”,
“_id”: “_ZRUZhJUSOySPk6ksqswBQ”,
“found”: false
}
and also we havent tried this opensearch 3.X.X

3

@krusha This would appear to be a bug, it might be related this. .

Would you be able to increase ManagedIndexCoordinator to DEBUG and provide the logs at the time when the new index is created?

You should be able to use the following:

PUT _cluster/settings
{
  "transient": {
    "logger.org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexCoordinator": "DEBUG"
  }
}
4

[2026-06-05T08:14:19,446][DEBUG][o.o.i.i.ManagedIndexCoordinator] [opensearch-k3-master-2] Couldn’t find any matching policy with appropriate permissions that can manage index k3-usage4-2026.05.29 this is logs showing

@Anthony And this index is recently created one and it has no policy attached:

{
“k3-usage4-2026.05.29”: {
“index.plugins.index_state_management.policy_id”: null,
“index.opendistro.index_state_management.policy_id”: null,
“enabled”: null
},
“total_managed_indices”: 0
}

5

@krusha It would appear that getPoliciesWithISMTemplates() in ManagedIndexCoordinator silently returns emptyList() on any SearchPhaseExecutionException or generic exception. When this happens during the sweepClusterChangedEvent (triggered by index creation), no managed index config is written. The self-healing background sweep() should recover this in 5 minutes, but if it encounters the same transient error on the same run, the index permanently misses the policy.

If the cluster was not overloaded at the time of the index creation, you can check the thread pool and heap usage using the following:

curl -ku admin:'admin' \     
  'https://localhost:9200/_nodes/opensearch-node/stats/thread_pool?filter_path=nodes.*.thread_pool.generic,nodes.*.thread_pool.write&pretty'

curl -ku admin:admin 'https://localhost:9200/_cat/nodes?v&h=name,heap.percent,heap.current,heap.max'

And the policy was attached to previous indices. I would recommend raising a bug issue for this here. With the debug message attached.

6

I dont see any overload and it seems to be normal

name heap.percent heap.current heap.max
opensearch-k3-ingest-0 8 259.3mb 3gb
opensearch-k3-master-2 9 288.7mb 3gb
opensearch-k3-master-1 19 584.5mb 3gb
opensearch-k3-data-1 64 6.4gb 10gb
opensearch-k3-data-5 61 6.1gb 10gb
opensearch-k3-data-2 40 4gb 10gb
opensearch-k3-data-3 67 6.7gb 10gb
opensearch-k3-master-0 13 400.7mb 3gb
opensearch-k3-data-4 59 5.9gb 10gb
opensearch-k3-data-0 66 6.6gb 10gb

Blockquote
and also
{
“nodes”: {
“LTN6HgCTR3m3ZdpZ-bGGXQ”: {
“thread_pool”: {
“generic”: {
“threads”: 10,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 10,
“completed”: 28851197
},
“write”: {
“threads”: 2,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 2,
“completed”: 128
}
}
},
“N1voWM5xSHSZ_3nOvei2pg”: {
“thread_pool”: {
“generic”: {
“threads”: 24,
“queue”: 0,
“active”: 1,
“rejected”: 0,
“largest”: 24,
“completed”: 28710147
},
“write”: {
“threads”: 2,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 2,
“completed”: 205
}
}
},
“vOazG9QKTM6NbwcodXCu0g”: {
“thread_pool”: {
“generic”: {
“threads”: 128,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 128,
“completed”: 28678923
},
“write”: {
“threads”: 2,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 2,
“completed”: 90
}
}
},
“1AyQal5tQu-wZ5Gwxa2nMg”: {
“thread_pool”: {
“generic”: {
“threads”: 41,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 41,
“completed”: 128464711
},
“write”: {
“threads”: 4,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 4,
“completed”: 3372064
}
}
},
“rPiAvWcuSNuFNQZHqXqFxw”: {
“thread_pool”: {
“generic”: {
“threads”: 39,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 39,
“completed”: 122960376
},
“write”: {
“threads”: 4,
“queue”: 0,
“active”: 2,
“rejected”: 0,
“largest”: 4,
“completed”: 4393251
}
}
},
“-3zu5tFESACqf3lfixskdw”: {
“thread_pool”: {
“generic”: {
“threads”: 44,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 44,
“completed”: 127330135
},
“write”: {
“threads”: 4,
“queue”: 0,
“active”: 1,
“rejected”: 0,
“largest”: 4,
“completed”: 3410445
}
}
},
“MBGj3fIYQgK-Ma6Dmpe13Q”: {
“thread_pool”: {
“generic”: {
“threads”: 28,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 28,
“completed”: 125985585
},
“write”: {
“threads”: 4,
“queue”: 0,
“active”: 2,
“rejected”: 0,
“largest”: 4,
“completed”: 3447912
}
}
},
“FNIwDJTwQHyYd8bZA5M3_A”: {
“thread_pool”: {
“generic”: {
“threads”: 55,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 55,
“completed”: 29156876
},
“write”: {
“threads”: 2,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 2,
“completed”: 277
}
}
},
“TB9ndrxvTKaQxn9EX7nDrA”: {
“thread_pool”: {
“generic”: {
“threads”: 36,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 36,
“completed”: 125221013
},
“write”: {
“threads”: 4,
“queue”: 0,
“active”: 1,
“rejected”: 0,
“largest”: 4,
“completed”: 3539787
}
}
},
“nR_G43RqRJOERyTIfSY9kg”: {
“thread_pool”: {
“generic”: {
“threads”: 33,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 33,
“completed”: 128222405
},
“write”: {
“threads”: 4,
“queue”: 0,
“active”: 0,
“rejected”: 0,
“largest”: 4,
“completed”: 3256635
}
}
}
}
}

@Anthony I will raise a bug ticket