Question: advantages of max_concurrent_shard_requests

OpenSearch
,
1

Describe the issue:

FYI : The default value of max_concurrent_shard_requests for _search API is 5.

For comparing between max_concurrent_shard_requests (5 vs 15), I picked up only 1% of users.

Surprisingly, though 1/100 API is called with the query parameter (max_concurrent_shard_requests=15), metrics for API latency seem to be much more stable in volatility and to have less up-and-down patterns than before.

image
image924×306 66.1 KB

But I’m curious of the reason that whether only 1% of _search APIs can improve performance. Does only few change in accessing concurrent shards make a big result?
(ex. High max_concurrent_shard_requests lets Query Cache to be stored faster.)


My situations:

  1. Almost every indices have heavy size documents in cluster:
green  open   xxx.reindex_2022.10 nHGTKkWGSzqi-k0I1pyBkg   8   2     829312         1181     38.4gb         12.8gb
green  open   xxx.reindex_2022.11 o0tI5rE-TEWKx_Jzoi2Xpg   8   2    1236982         2165     72.4gb         24.1gb
green  open   xxx.reindex_2022.12 TdTiyJHDSZubYhi-Pjrtuw   8   2    1963307         1415      170gb         56.6gb
green  open   xxx.reindex_2023.01 dNY1rAoFRUSj1cuX-jyqOg   8   2    1201577         1632     51.8gb         17.2gb
...
green  open   xxx.reindex_2024.05 WlhutjMARg6SBWyoABY2zw   8   2    1029606        11817     32.4gb         10.8gb
green  open   xxx.reindex_2024.06 lqAWgna7Sc6x5uMzr3JxXw   8   2     953041       109980     27.8gb          9.2gb
green  open   xxx.reindex_2024.07 -lS3sKEHQQu2HusEawF-UQ   8   2    1473063       121489     24.6gb            8gb

  1. Status of cluster:
  • cpu: 12vCore
  • jvm : 32GB (each Data Node)
  • The number of primary shard : 1
  • The number of replica shard : 2 (for reducing _search API latency)
Documents : 104625405
Disk Usage : 7.3 TB
Primary Shards : 1989
Replica Shards  : 3941