Troubleshoot hotthreads

[infodata]

anyway to check what index is from hotthreads?
write
refresh
Lucene Merge Thread

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
“version” : {
“distribution” : “opensearch”,
“number” : “2.2.0”,

Describe the issue:
how to check issues on hotthreads find index name

Configuration:

Relevant Logs or Screenshots:

::: {opensearch-node1}{xZUY_1tBQO2sk771fBhFLA}{lf6YPYMxQmmsXzpU2v6EcA}{172.18.0.2}{172.18.0.2:9300}{dimr}{shard_indexing_pressure_enabled=true}
Hot threads at 2023-03-14T15:00:27.836Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

6.1% (30.6ms out of 500ms) cpu usage by thread 'opensearch[opensearch-node1][write][T#2]'
 10/10 snapshots sharing following 2 elements
   java.base@17.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   java.base@17.0.4/java.lang.Thread.run(Thread.java:833)

5.1% (25.6ms out of 500ms) cpu usage by thread 'opensearch[opensearch-node1][generic][T#6]'
 2/10 snapshots sharing following 35 elements
   app//org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:193)
   app//org.apache.lucene.index.IndexingChain$PerField.invert(IndexingChain.java:1224)
   app//org.apache.lucene.index.IndexingChain.processField(IndexingChain.java:729)
   app//org.apache.lucene.index.IndexingChain.processDocument(IndexingChain.java:620)
   app//org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:241)
   app//org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:432)
   app//org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1532)
   app//org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1817)
   app//org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1470)
   app//org.opensearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1270)
   app//org.opensearch.index.engine.InternalEngine.indexIntoLucene(InternalEngine.java:1206)
   app//org.opensearch.index.engine.InternalEngine.index(InternalEngine.java:1009)
   app//org.opensearch.index.shard.IndexShard.index(IndexShard.java:965)
   app//org.opensearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:910)
   app//org.opensearch.index.shard.IndexShard.applyTranslogOperation(IndexShard.java:1762)
   app//org.opensearch.index.shard.IndexShard.runTranslogRecovery(IndexShard.java:1817)
   app//org.opensearch.index.shard.IndexShard.lambda$openEngineAndRecoverFromTranslog$9(IndexShard.java:1865)
   app//org.opensearch.index.shard.IndexShard$$Lambda$4072/0x0000000801957a98.run(Unknown Source)
   app//org.opensearch.index.engine.InternalEngine.recoverFromTranslogInternal(InternalEngine.java:568)
   app//org.opensearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:543)
   app//org.opensearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:145)
   app//org.opensearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1874)
   app//org.opensearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:500)
   app//org.opensearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:112)
   app//org.opensearch.index.shard.StoreRecovery$$Lambda$4071/0x0000000801956f30.get(Unknown Source)
   app//org.opensearch.action.ActionListener.completeWith(ActionListener.java:342)
   app//org.opensearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:110)
   app//org.opensearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:2176)
   app//org.opensearch.index.shard.IndexShard$$Lambda$4066/0x0000000801955cc8.accept(Unknown Source)
   app//org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:88)
   app//org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806)
   app//org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
   java.base@17.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
   java.base@17.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
   java.base@17.0.4/java.lang.Thread.run(Thread.java:833)

[radu.gheorghe]

Hi,

You can’t check which index is affected just by looking at hot threads. Buuut, if you know the workload - in your case, write (i.e. index), refresh, merge - then you can check the metrics over time for the respective workloads, which are exposed per-index via Indices Stats API.

Sematext Cloud can get those metrics for you out of the box (disclaimer: I work for Sematext). Here’s an example specifically for refreshes from the default Index Stats dashboard: