OpenSearch Hot-Warm-Cold Architecture Not Functioning in Kubernetes Cluster

Dear Team,

We have deployed an OpenSearch cluster with the following configuration:

Hot Phase: 2 pods (StatefulSet 1)

Warm/Cold Phase: 2 pods (StatefulSet 2)

Both StatefulSets are running in the same namespace.

The cluster deployment is successful, but we are encountering an issue when applying the Index State Management (ISM) policy to transition indices from the hot phase to the warm phase. Despite the policy being applied, the indices remain in the hot phase PVCs and do not migrate to the warm phase as expected.

We have apply the below ISM policy:

*##################
PUT _plugins/_ism/policies/hot_warm_cold_delete
{
“policy”: {
“description”: “Hot (1h) → Warm (7d) → Cold (90d) → Delete”,
“default_state”: “hot”,
“states”: [
{
“name”: “hot”,
“actions”: [
{
“replica_count”: {
“number_of_replicas”: 1
}
},
{
“index_priority”: {
“priority”: 100
}
}
],
“transitions”: [
{
“state_name”: “warm”,
“conditions”: {
“min_index_age”: “1h”
}
}
]
},
{
“name”: “warm”,
“actions”: [
{
“replica_count”: {
“number_of_replicas”: 1
}
},
{
“force_merge”: {
“max_num_segments”: 1
}
},
{
“allocation”: {
“include”: {
“temperature”: “warm”
}
}
},
{
“index_priority”: {
“priority”: 50
}
}
],
“transitions”: [
{
“state_name”: “cold”,
“conditions”: {
“min_index_age”: “7d”
}
}
]
},
{
“name”: “cold”,
“actions”: [
{
“read_only”: {}
},
{
“replica_count”: {
“number_of_replicas”: 0
}
},
{
“allocation”: {
“include”: {
“data”: “cold”
}
}
},
{
“index_priority”: {
“priority”: 0
}
}
],
“transitions”: [
{
“state_name”: “delete”,
“conditions”: {
“min_index_age”: “90d”
}
}
]
},
{
“name”: “delete”,
“actions”: [
{
“delete”: {}
}
]
}
],
“ism_template”: {
“index_patterns”: [“hotindex”],
“priority”: 100
}
}
}

###################**

Please help us to get this work , please let us know if any additional config required from our end.

Thanks,
Anil

@anilmca01 What do you see when you run explain API against the index?

Did you set node.attr.temperature: warm in your warm node?

Hi @pablo ,

We have configured the node.attr.temperature: warm value in the StatefulSet configuration for our warm pods.

Regarding the Explain API, we are not seeing any output because our cluster status is currently “GREEN.” The Explain API would only provide relevant output if there were unassigned shards in the cluster.

In our current setup:

  • We are using two separate storage classes for hot and warm nodes in our StatefulSets.

  • ISM can facilitate this transition by automatically moving indices from hot nodes (with one storage class) to warm nodes (with another storage class) based on defined policies.

  • But indices were not moving from hot to warm, but indices states were changed from hot to warm.

  • Please let us know is it possible or not?

Thanks

Anil

@anilmca01 The ISM explain API will tell you if the policy has kicked in and initiated the warm phase.