[Solved] Is there document or any reference for tuning the recovery performance?

OpenSearch
1

We’re adding 40 data nodes to an existing cluster. There’re many shards that need be relocated.
No Recovery tuning yet. I can see there’s recovery setting for ELK Indices Recovery | Elasticsearch Guide [2.3] | Elastic
May I know if there’s an equivalent document for OpenSearch?

Problems:

  • Only 2 shards are relocating in parallel. I believe this can be tunned.
  • The speed is limited to 40MB/s for NVME drives. I guess it’s throttled by the recovery setting.

Thanks for the advice.

Hugo

2

The transfer speed is faster with the change. But still, only two shards with stage index. Can’t find document to increase the concurrent index recover shards.

Screen Shot 2022-04-27 at 8.57.13 PM
Screen Shot 2022-04-27 at 8.57.13 PM1032×212 31.9 KB 

PUT _cluster/settings
{
    "persistent" : {
        "indices.recovery.max_bytes_per_sec" : "500mb",
        "indices.recovery.max_concurrent_file_chunks": 5,
        "indices.recovery.max_concurrent_operations" : 4
        
    }
}
3

Use the max_bytes_sec setting you have above and the following:

cluster.routing.allocation.cluster_concurrent_rebalance
cluster.routing.allocation.node_concurrent_recoveries
1 Like
4

[SOLVED]
@mrweber That works well. You save my day. How do you know these settings? Are there somewhere in OpenSearch document?

Screen Shot 2022-04-28 at 2.43.57 PM
Screen Shot 2022-04-28 at 2.43.57 PM886×242 20.8 KB

Screen Shot 2022-04-28 at 2.43.07 PM
Screen Shot 2022-04-28 at 2.43.07 PM845×774 66.1 KB

For who is looking for answer in the future. This is the setting I’m using to boost the recovery speed. You’ll want to lower the max_bytes_per_sec while the cluster expansion is completed.

PUT _cluster/settings
{
    "persistent" : {
        "indices.recovery.max_bytes_per_sec" : "500mb",
        "indices.recovery.max_concurrent_file_chunks": 5,
        "indices.recovery.max_concurrent_operations" : 4,
        "cluster.routing.allocation.node_concurrent_recoveries": 6,
        "cluster.routing.allocation.cluster_concurrent_rebalance": 6
    }
}

Thanks // Hugo

5

I just know these from years of working with elasticsearch. You can still find relevant information in the elasticsearch 7.10 docs.

Thanks,
Matt Weber

3 Likes