# Open Search cluster is running high cpu and response time is also high
**URL:** https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111
**Category:** OpenSearch
**Tags:** discuss
**Created:** [October 2, 2023, 11:35pm UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111 "2023-10-02T23:35:34Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@imdbtoimdb](https://forum.opensearch.org/u/imdbtoimdb)
#### Post date: [October 2, 2023, 11:35pm UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/1 "2023-10-02T23:35:34Z")
**Versions** (relevant - OpenSearch/Dashboard/Server OS/Browser):
version" : {
“distribution” : “opensearch”,
“number” : “1.2.2”,
“build\_type” : “tar”,
“build\_hash” : “123d41ce4fad54529acd7a290efed848e707b624”,
“build\_date” : “2021-12-15T18:03:07.761961Z”,
“build\_snapshot” : false,
“lucene\_version” : “8.10.1”,
“minimum\_wire\_compatibility\_version” : “6.8.0”,
“minimum\_index\_compatibility\_version” : “6.0.0-beta1”
},
**Describe the issue** :
Hello All,
We recently migrated our search cluster from ES ( “version” : {
“number” : “6.8.16”,
“build\_flavor” : “default”,
“build\_type” : “deb”,
“build\_hash” : “1f62092”,
“build\_date” : “2021-05-21T19:27:57.985321Z”,
“build\_snapshot” : false,
“lucene\_version” : “7.7.3”,
“minimum\_wire\_compatibility\_version” : “5.6.0”,
“minimum\_index\_compatibility\_version” : “5.0.0”
},
to open search . Wrt the H/W both are same . But we are seeing high cpu spike in open search cluster . not sure what are all the next step . Any help would be appreciated
**Configuration** :
No of shards : 24 on both OS and ES ( replica count is 1 per primary shard ) .
**Relevant Logs or Screenshots** :
---
### Author:  [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)
#### Post date: [October 4, 2023, 1:13pm UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/2 "2023-10-04T13:13:41Z")
I think the next step is to [monitor OpenSearch](https://sematext.com/docs/integration/opensearch/) and see what’s taking CPU. It could be [GC](https://sematext.com/blog/java-garbage-collection-tuning/), it could be indexing, queries or maybe some other thread pool that’s doing work. We wrote a [metrics guide for Elasticsearch](https://sematext.com/blog/top-10-elasticsearch-metrics-to-watch/) a while ago that mostly applies to OpenSearch.
---
### Author:  [@imdbtoimdb](https://forum.opensearch.org/u/imdbtoimdb)
#### Post date: [October 5, 2023, 6:18pm UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/3 "2023-10-05T18:18:41Z")
Thanks for the kind reply. After analyze the profile output seeing most of the call taking more time in the TermQuery . Any suggestions to fix this one ?
Thanks
---
### Author:  [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)
#### Post date: [October 6, 2023, 5:04am UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/4 "2023-10-06T05:04:20Z")
TermQuery is the most basic query that can run, you can’t really optimize that.
Actually you can (with a more aggressive [merge policy](https://sematext.com/blog/solr-optimize-is-not-bad-for-you-lucene-solr-revolution/) - the blog post is about Solr, but you have similar options in OpenSearch) but usually the problem is higher up. For example, the number of TermQuery clauses, the layout of your data, number of shards, how well they’re balanced, etc.
---
### Author:  [@imdbtoimdb](https://forum.opensearch.org/u/imdbtoimdb)
#### Post date: [October 9, 2023, 11:00pm UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/5 "2023-10-09T23:00:57Z")
Thanks for the kind response . One thing we are seeing compare to ES to OS is from the Hardware perspective both are same machines . But in OS profiler data shows the term query section taking ~240ms but the same data ES is taking ~92 ms . Hence not sure what is the problem . Here both shards are same ( 16 ) and 8 data nodes
---
### Author:  [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)
#### Post date: [October 10, 2023, 8:10am UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/6 "2023-10-10T08:10:32Z")
I don’t know why you’d see that difference besides:
- the “random” distribution of documents between shards, if you’re letting OS/ES chose IDs
- the “random” nature of merges, when they kick in
- the Lucene version
And you can’t do much about any of the above. Which is why I’d generally suggest concentrating on optimizing what you have vs comparing to what you had before. Unless you’re still deciding whether to make the upgrade or not.
---
### Author:  [@imdbtoimdb](https://forum.opensearch.org/u/imdbtoimdb)
#### Post date: [October 19, 2023, 3:14am UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/7 "2023-10-19T03:14:28Z")
thanks for the reply . i have increased the shards count to 32 and seeing better performance . Any idea how this shards count plays major role here . still not able to connect these dots .
---
### Author:  [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)
#### Post date: [October 26, 2023, 2:17pm UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/8 "2023-10-26T14:17:45Z")
With more shards you’re parallelizing queries more. But there’s also more overhead in merging per-shard results.
Maybe concurrent segment search will help you? [Introducing concurrent segment search in OpenSearch · OpenSearch](https://opensearch.org/blog/concurrent_segment_search/)
---
### Author:  [@imdbtoimdb](https://forum.opensearch.org/u/imdbtoimdb)
#### Post date: [November 3, 2023, 12:22am UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/9 "2023-11-03T00:22:48Z")
interesting fact is i created a new cluster with same data ( created a new index and back-filled the data ) . but now i am seeing how response time . Any specific warm ups / warm up period required ? its really super surprising now
---
### Author:  [@radu.gheorghe](https://forum.opensearch.org/u/radu.gheorghe)
#### Post date: [November 3, 2023, 5:29am UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/10 "2023-11-03T05:29:38Z")
Yeah, if you run a query right after ingesting a lot of data, it might be that the OS cache doesn’t have everything it needs in the page cache. Plus, all Elasticsearch-specific query-related caches (query cache, request cache) will be cold.
---
### Author:  [@imdbtoimdb](https://forum.opensearch.org/u/imdbtoimdb)
#### Post date: [November 6, 2023, 3:19am UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/11 "2023-11-06T03:19:06Z")
Thanks. is there any way can we improve the things? since we didnt use the k-nn option while create the index since its prod index .
---
### Author:  [@imdbtoimdb](https://forum.opensearch.org/u/imdbtoimdb)
#### Post date: [November 9, 2023, 7:38pm UTC](https://forum.opensearch.org/t/open-search-cluster-is-running-high-cpu-and-response-time-is-also-high/16111/12 "2023-11-09T19:38:46Z")
Adding more updates :
Here we added query cache to 20% which helped little bit . But still the cluster CPU is not lowering . On the other side the Elastic search cluster which handling the load very efficient. Any more recommendations would be great ?