ISM policy: index moves from hot state to warm state does not get delete and assigned to the correct node

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:

I am using the time-based index pattern with ISM policy set up.

I current have 5 data node, 3 hot node (os1, 2, 3) and 2 warm node (os4, os5). My goal is the index in hot state after 14 days can be shrink and force merge and move to the warm state with 1 replica and 1 primary shard.

Observation:

When the index reaches to the 14 days age, it created a shrunked_index, but the original index is still there and does not get deleted, and the newly create shrunked_index is in os1 and os 2 with 1 replica and 1 primary but no in the os4 and os5.

Configuration:

This my node set up:

node attr                            value
os2  zone                            zoneB
os2  temp                            hot
os2  shard_indexing_pressure_enabled true
os4  zone                            zoneA
os4  temp                            warm
os4  shard_indexing_pressure_enabled true
os1  temp                            hot
os1  zone                            zoneA
os1  shard_indexing_pressure_enabled true
os3  zone                            zoneC
os3  temp                            hot
os3  shard_indexing_pressure_enabled true
os5  zone                            zoneB
os5  temp                            warm
os5  shard_indexing_pressure_enabled true

This is my ISM Policy set up :

PUT _plugins/_ism/policies/mobile_policy
{
  "policy": {
    "policy_id": "mobile_policy",
    "description": "Policy to manage mobile indices with hot-warm, shrink, force merge, snapshot, and delete",
    "default_state": "hot",
    "schema_version": 1,
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "rollover": {
              "min_index_age": "1d"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "warm",
            "conditions": {
              "min_index_age": "14d"
            }
          }
        ]
      },
      {
        "name": "warm",
        "actions": [
          {
            "read_only": {}
          },
          {
            "shrink": {
              "num_new_shards": 1,
              "target_index_name_template": {
                "source": "shrunken-{{ctx.index}}" 
              }
            }
          },
          {
            "force_merge": {
              "max_num_segments": 1
            }
          },
          {
            "replica_count": {
              "number_of_replicas": 1
            }
          },
          {
            "allocation": {
              "require": {
                "temp": "warm"
              }
            }
          }
        ],
    "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "365d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "snapshot": {
              "repository": "mobile-na-s3-repository",
              "snapshot": "snapshot-{{ctx.index}}" 
            }
          },  {
            "delete": {}
          }
        ]
      }
    ],
    "ism_template": [
      {
        "index_patterns": ["mobile_na*"],
        "priority": 100
      }
    ]
  }
}

, and this is my index template



PUT _index_template/mobile_template
{
  "index_patterns": ["mobile_na-*"],
  "template": {
    "settings": {
      "index": {
        "number_of_shards": 1,
        "number_of_replicas": 0,
          "plugins.index_state_management.policy_id": "mobile_policy",
        "plugins.index_state_management.rollover_alias": "mobile_na"
      }
    },
    "mappings": {
    
      "dynamic_templates": [
        {
          "name_fields_as_keyword": {
            "match": "rx*",
            "mapping": {
              "type": "keyword"
            }
          }
        }
      ],
      "properties": {
        "timestamp": {
          "type": "date",
          "format": "epoch_second"
        },
        "parsedAt": {
          "type": "date",
          "format": "epoch_millis"
        }
      }
    }
  },
  "priority": 100
}

Could anyone help me out?

2

fixed. just switched the order of the action in warm state.