We recently implemented a shrink state in our index policy to optimize storage post index roll-over and noticed something interesting, Once the index is “shrunk” (down from N+1 shards to 1+1) the new index retains a setting:
"routing": { -
"allocation": { -
"initial_recovery": { -
"_id": "Qldd4ZvkSKGqfU-h3jZgbw"
}
}
},
Which always forces the new primary and its replica onto the same two nodes, over the last few days none of these shards have re-balanced to other nodes and because of it I have a very un-even storage utilization on two nodes in comparison to the rest of the cluster.
Is this expected?
Are there any settings I am missing to remove that routing.allocation.initial_recovery setting so the shards can re-distribute?
Thanks.
(Full ISM policy below)
{
"id": "rollover",
"seqNo": 34459949,
"primaryTerm": 38,
"policy": {
"policy_id": "rollover",
"description": "A sample description of the policy",
"last_updated_time": 1709327935258,
"schema_version": 21,
"error_notification": {
"channel": {
"id": "cZICrYEB_9J1zuttzC3B"
},
"message_template": {
"source": "The index {{ctx.index}} failed during policy execution.",
"lang": "mustache"
}
},
"default_state": "rollover",
"states": [
{
"name": "rollover",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"rollover": {
"min_size": "100gb",
"min_index_age": "1d",
"copy_alias": false
}
}
],
"transitions": [
{
"state_name": "reduce_replicas"
}
]
},
{
"name": "reduce_replicas",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"replica_count": {
"number_of_replicas": 0
}
}
],
"transitions": [
{
"state_name": "shrink_shards"
}
]
},
{
"name": "shrink_shards",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"shrink": {
"num_new_shards": 1,
"target_index_name_template": {
"source": "{{ctx.index}}_shrunken",
"lang": "mustache"
},
"aliases": [
{
"logstash": {
"is_write_index": null
}
}
],
"switch_aliases": true,
"force_unsafe": true
}
}
],
"transitions": [
{
"state_name": "delete_multi_primary"
}
]
},
{
"name": "delete_multi_primary",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"delete": {}
}
],
"transitions": []
}
],
"ism_template": [
{
"index_patterns": [
"logstash-*"
],
"priority": 1,
"last_updated_time": 1656463404326
}
]
}
}