Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
3.6.0 (Upgraded from 1.3 → 2.19 → 3.6.0)
Describe the issue:
After last upgrade 2.19 → 3.6 we noted that our ISM policy is not working anymore. We are shifting our hot indices after 7 days to warm node (no replica anymore) and after 30 days we delete them.
But since last upgrade the indices stay on hot node. I dont see any problems in logfile.
Is there any possibility to analyze the problem? Any debug settings?
We did reindex of all system indices and created aliases. Maybe this could cause the problem?
@rofr_dd This appears to be a bug, and I would recommend to raise a issue here
As a temporary workaround, you manually reply the policy the indices using the following:
POST _plugins/_ism/remove/<index>
then
POST _plugins/_ism/add/<index>
The issue seems to be happening when the policy is updated, I used the following steps to reproduce, you can include these in the issue too:
-
Single-node 3.6.0 cluster (opensearchproject/opensearch:3.6.0).
-
Create policy P (becomes seq_no=N):
hot → (after min_index_age) → warm, where warm runs replica_count: { number_of_replicas: 0 }, attached via ism_template.
-
Create an index matching P’s index_patterns (e.g. 2 replicas).
-
Update P (now seq_no=N+2) so warm instead runs replica_count: { number_of_replicas: 3 }.
-
Wait for the index to transition hot → warm and run its action.
-
Observe:
-
_plugins/_ism/explain/<index> still reports policy_seq_no: N — never advanced, despite multiple coordinator sweeps and job runs in between.
-
The action log reads "Successfully set number_of_replicas to 0" — the old value.
-
GET <index>/_settings confirms index.number_of_replicas: 0, not 3.
I figured out after reindexing an alias was created which is the cause of the problem. When doing an additional reindex back to the old name (remove alias before) solved the problem.