Data node thread pool

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue: Certain OpenSearch data nodes are becoming overloaded due to heavy search execution. The overload is visible through elevated search thread activity, rejected search requests, and latency spikes. Impacting overall latency

Relevant Logs or Screenshots:

Screenshot 2026-07-01 at 10.37.28 AM
Screenshot 2026-07-01 at 10.37.28 AM1484×618 34.5 KB

2

@aussie Welcome to the forum!

The spikes in your Thread Pool Search Queue mean search requests are arriving faster than the available search threads can process them. When the queue fills up (default depth: 1000), OpenSearch starts rejecting requests outright with an OpenSearchRejectedExecutionException.

Pull per-node thread pool stats to identify which nodes are saturated (Nodes stats API):

GET /_nodes/stats/thread_pool

Check search.rejected, search.queue, and search.active on each node. Then look at what’s actually running during spikes (Hot threads API):

GET /_nodes/hot_threads

Enable the slow log to surface expensive queries:

PUT /your-index/_settings
{
  "index.search.slowlog.threshold.query.warn": "5s",
  "index.search.slowlog.threshold.query.info": "2s"
}

Common causes and fixes:

  • Shard imbalance: some nodes hold far more (or heavier) shards than others, so search traffic is unevenly distributed. Check with GET /_cat/shards?v and GET /_cat/allocation?v.

  • Expensive queries: wildcard-leading patterns, large from+size pagination, script scoring, or heavy aggregations. The slow log will surface these.

  • Too many shards per node: each query fans out to every shard on a node. Reducing shard count (by shrinking or merging indices) lowers per-query overhead significantly.

  • High segment counts on frequently queried indices: force merging reduces per-shard work:

    POST /your-index/_forcemerge?max_num_segments=1

Increasing queue depth (buys headroom during bursts, doesn’t fix throughput):

This setting is dynamic, no restart needed (Thread pool settings):

PUT /_cluster/settings
{
  "persistent": {
    "thread_pool.search.queue_size": 2000
  }
}

If you want to increase the thread count itself (thread_pool.search.size), that requires a node restart. The default is ((available_processors × 3) / 2) + 1.

The spike pattern in your graph looks like burst traffic rather than sustained overload, I’d start by identifying the source (dashboards, scheduled jobs, a specific API caller) before scaling anything.

Hope this helps

3

Thanks @Anthony We are reviewing this internally and will let you know if we make any changes.

4

@anthony We have 12 data nodes, and the issue occurs on random data nodes. Here are the CPU metrics and hot threads we captured from one of the impacted data node.

{opensearch-green-cluster-data-2}{ILqRxKcDRmCJnlz-8D23Sg}{TtzqfXv7QzmqjN_LmBiAuQ}{dir}{color=green, shard_indexing_pressure_enabled=true}
Hot threads at 2026-07-01T18:28:34.693Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

66.8% (334.1ms out of 500ms) cpu usage by thread ‘opensearch[opensearch-green-cluster-data-2][search][T#25]’
3/10 snapshots sharing following 28 elements
app//org.opensearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:289)
app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560)
app//org.opensearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:355)
app//org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:462)
app//org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWithCollector(QueryPhase.java:450)
app//org.opensearch.search.query.QueryPhase$DefaultQueryPhaseSearcher.searchWith(QueryPhase.java:432)
app//org.opensearch.search.query.QueryPhaseSearcherWrapper.searchWith(QueryPhaseSearcherWrapper.java:60)
org.opensearch.neuralsearch.search.query.HybridQueryPhaseSearcher.searchWith(HybridQueryPhaseSearcher.java:61)
app//org.opensearch.search.query.QueryPhase.executeInternal(QueryPhase.java:282)
app//org.opensearch.search.query.QueryPhase.execute(QueryPhase.java:155)
app//org.opensearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:648)
app//org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:712)
app//org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:681)
app//org.opensearch.search.SearchService$2$$Lambda/0x00007935654dc8a0.get(Unknown Source)
app//org.opensearch.search.SearchService$$Lambda/0x00007935654dcab8.get(Unknown Source)
app//org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74)
app//org.opensearch.action.ActionRunnable$$Lambda/0x00007935653879d8.accept(Unknown Source)
app//org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89)
app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
app//org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78)
app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
app//org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59)
app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1014)
app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
java.base@21.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
java.base@21.0.6/java.lang.Thread.runWith(Thread.java:1596)
java.base@21.0.6/java.lang.Thread.run(Thread.java:1583)

image
image448×286 13 KB

5

@aussie The hot threads output identifies the root cause, it would appear you’re running hybrid queries via the neural-search plugin (HybridQueryPhaseSearcher). These are significantly more CPU-intensive than standard queries by design: on each shard, they execute multiple sub-queries (neural/k-NN vector search + BM25), then normalize and combine the scores before returning results. That’s the CPU you’re seeing on those search threads.

The random-node pattern and spike shape suggest bursts of concurrent hybrid queries rather than a sustained throughput problem, so the first thing I’d do is confirm what’s triggering them: a dashboard, application, or scheduled pipeline.

Tuning options:

  1. Lower ef_search: this directly reduces the number of candidate vectors evaluated per shard during HNSW graph traversal, which is where the CPU is going. The default is already 100, so you need to go below that to see any benefit. Test with 50 as a starting point:
{
  "neural": {
    "passage_embedding": {
      "query_text": "...",
      "model_id": "...",
      "k": 50,
      "method_parameters": {
        "ef_search": 50
      }
    }
  }
}

There will be a recall trade-off, so measure against your relevance benchmarks before going lower.

  1. Reduce shard count on these indices: the full hybrid pipeline (both sub-queries + normalization) runs independently on every shard. Fewer shards per node = less total work per query.

Hope this helps

6

@Anthony : Working with @aussie on this.

Thanks for your inputs.

  1. Firstly, yes, reducing ef_search is going to have a direct impact on recall which isnt an option

we are currently using ef_search: 170 with combination of min_score (minscore is dynamic for each query). so tuning this wouldnt be helpful.

  1. our index has 4 shards which is read/write balanced as per recommendation (~60 gb index).
Our Current config:

    a) 4 primary shards
    b) 12 replicas
    c) Distributed across 12 nodes
So here, each node has these heavy duty 4 shards.

Question: 

A) Is it better to reduce the number of replicas like
  4 replicas across 12 nodes : so shards per nodes reduces with more compute and memory headroom for normal and knn computes?

B) Can this improve the performance with same kind of load, with more efficient parallel reads/writes?

Thanks!