[Solved] Is there document or any reference for tuning the recovery performance?

We’re adding 40 data nodes to an existing cluster. There’re many shards that need be relocated.
No Recovery tuning yet. I can see there’s recovery setting for ELK Indices Recovery | Elasticsearch Guide [2.3] | Elastic
May I know if there’s an equivalent document for OpenSearch?

Problems:

  • Only 2 shards are relocating in parallel. I believe this can be tunned.
  • The speed is limited to 40MB/s for NVME drives. I guess it’s throttled by the recovery setting.

Thanks for the advice.

Hugo

The transfer speed is faster with the change. But still, only two shards with stage index. Can’t find document to increase the concurrent index recover shards.

PUT _cluster/settings
{
    "persistent" : {
        "indices.recovery.max_bytes_per_sec" : "500mb",
        "indices.recovery.max_concurrent_file_chunks": 5,
        "indices.recovery.max_concurrent_operations" : 4
        
    }
}

Use the max_bytes_sec setting you have above and the following:

cluster.routing.allocation.cluster_concurrent_rebalance
cluster.routing.allocation.node_concurrent_recoveries
1 Like

[SOLVED]
@mrweber That works well. You save my day. How do you know these settings? Are there somewhere in OpenSearch document?


For who is looking for answer in the future. This is the setting I’m using to boost the recovery speed. You’ll want to lower the max_bytes_per_sec while the cluster expansion is completed.

PUT _cluster/settings
{
    "persistent" : {
        "indices.recovery.max_bytes_per_sec" : "500mb",
        "indices.recovery.max_concurrent_file_chunks": 5,
        "indices.recovery.max_concurrent_operations" : 4,
        "cluster.routing.allocation.node_concurrent_recoveries": 6,
        "cluster.routing.allocation.cluster_concurrent_rebalance": 6
    }
}

Thanks // Hugo

I just know these from years of working with elasticsearch. You can still find relevant information in the elasticsearch 7.10 docs.

Thanks,
Matt Weber

2 Likes