Openseach too many shards might cause heavy CPU usage

General Feedback
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): AWS managed cluster 2.19

Describe the issue:

Cluster design -

3 years data with splitting into monthly indices

  • xx-2025-09, xx-2025-08,…

2 current usage,

  • user search for specific month index only ..hot index on current 1-3 months
  • user search across all 3 years data which mean query using alias for 36 months index with total 220 shards

I did a test with opensearch benchmark and found out 2-3 request per second for search across 3 years spiking up the CPU very high >80% - 100%.

I believe because query searching for 220 shards make the OS cluster CPU hard.

Wondering with 30 index with around 80-90GB per index with 3 primary and 1 replica and average 25GB per shards

Should I reduce number of shards for old indices only since less write and usage ( so that this mean shards with bigger size. Thinking

  • Old indices like 2024-2023-2022
    • 1 shard = 100GB? out of recommendation for 10-30 GB per shard)
    • 2 shards - 40 GB
  • Hot indices remains
    • 25 GB per shard for 3 primary

Anyone with past experience able to advice should I try out 1 shard 100GB or 2 shard 40GB to reduce number of shards

Configuration:

Relevant Logs or Screenshots: