Openseach too many shards might cause heavy CPU usage

General Feedback
,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): AWS managed cluster 2.19

Describe the issue:

Cluster design -

3 years data with splitting into monthly indices

  • xx-2025-09, xx-2025-08,…

2 current usage,

  • user search for specific month index only ..hot index on current 1-3 months
  • user search across all 3 years data which mean query using alias for 36 months index with total 220 shards

I did a test with opensearch benchmark and found out 2-3 request per second for search across 3 years spiking up the CPU very high >80% - 100%.

I believe because query searching for 220 shards make the OS cluster CPU hard.

Wondering with 30 index with around 80-90GB per index with 3 primary and 1 replica and average 25GB per shards

Should I reduce number of shards for old indices only since less write and usage ( so that this mean shards with bigger size. Thinking

  • Old indices like 2024-2023-2022
    • 1 shard = 100GB? out of recommendation for 10-30 GB per shard)
    • 2 shards - 40 GB
  • Hot indices remains
    • 25 GB per shard for 3 primary

Anyone with past experience able to advice should I try out 1 shard 100GB or 2 shard 40GB to reduce number of shards

Configuration:

Relevant Logs or Screenshots:

2

Hi @kitkit ,

Searching over a large number of shards will yield results like you’re seeing.

If you plan to change your design to create better searches, I would first consider a few items.

I would not increase the total shard size to 100GB, even if for old indices that will be searched less, as when searched they may leave you with bigger CPU issues.

I would read this - Choosing the number of shards - Amazon OpenSearch Service

Let us know if you try 40GB per shard, and what results you see.

Leeroy.