Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
Observed in OpenSearch 1.3.16 as well as 1.2.3
Describe the issue:
We have a benchmark test which does parallel indexing, refresh indices and different types of query at the same time. It is observed that on an average OpenSearch is taking 30% more CPU on JDK11 as compared to JDK8. Has anyone observed this?
Configuration:
Relevant Logs or Screenshots:
From top command out that gives thread wise CPU consumption, following threads are detected
2076043 search-user 20 0 9866880 2.4g 60792 R 87.5 5.0 0:15.67 2075480 opensearch[custom_
2076032 search-user 20 0 9866880 2.4g 60792 R 81.2 5.0 0:15.60 2075480 opensearch[custom_
2076040 search-user 20 0 9866880 2.4g 60792 R 81.2 5.0 0:15.06 2075480 opensearch[custom_
2076044 search-user 20 0 9866880 2.4g 60792 R 87.5 5.0 0:13.92 2075480 opensearch[custom_
2097763 search-user 20 0 9866880 2.4g 60792 R 87.5 5.0 0:02.55 2075480 opensearch[custom_
2097791 search-user 20 0 9866880 2.4g 60792 R 18.8 5.0 0:00.03 2075480 opensearch[custom_
Stacktraces:
2076043 search-user 20 0 9866880 2.4g 60792 R 87.5 5.0 0:15.67 2075480 opensearch[custom_
“opensearch[custom_node][write][T#11]” #107 daemon prio=5 os_prio=0 cpu=15777.39ms elapsed=436.76s tid=0x000067be4414f800 nid=0x1fad8b waiting on condition [0x000067bd931d8000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.22/Native Method)
- parking to wait for <0x000067bf3d86e130> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.xfer(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.take(java.base@11.0.22/Unknown Source)
at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:176)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.22/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.22/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.22/Unknown Source)
at java.lang.Thread.run(java.base@11.0.22/Unknown Source)
2076032 search-user 20 0 9866880 2.4g 60792 R 81.2 5.0 0:15.60 2075480 opensearch[custom_
“opensearch[custom_node][write][T#5]” #100 daemon prio=5 os_prio=0 cpu=15632.62ms elapsed=437.82s tid=0x000067be4413f800 nid=0x1fad80 waiting on condition [0x000067be144a9000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.22/Native Method)
- parking to wait for <0x000067bf3d86e130> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.xfer(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.take(java.base@11.0.22/Unknown Source)
at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:176)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.22/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.22/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.22/Unknown Source)
at java.lang.Thread.run(java.base@11.0.22/Unknown Source)
2076040 search-user 20 0 9866880 2.4g 60792 R 81.2 5.0 0:15.06 2075480 opensearch[custom_
“opensearch[custom_node][write][T#10]” #106 daemon prio=5 os_prio=0 cpu=15096.36ms elapsed=436.88s tid=0x000067be28137800 nid=0x1fad88 waiting on condition [0x000067bd93361000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.22/Native Method)
- parking to wait for <0x000067bf3d86e130> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.xfer(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.take(java.base@11.0.22/Unknown Source)
at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:176)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.22/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.22/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.22/Unknown Source)
at java.lang.Thread.run(java.base@11.0.22/Unknown Source)
2076044 search-user 20 0 9866880 2.4g 60792 R 87.5 5.0 0:13.92 2075480 opensearch[custom_
“opensearch[custom_node][write][T#12]” #108 daemon prio=5 os_prio=0 cpu=13999.71ms elapsed=436.76s tid=0x000067be44151800 nid=0x1fad8c waiting on condition [0x000067bd93061000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.22/Native Method)
- parking to wait for <0x000067bf3d86e130> (a java.util.concurrent.LinkedTransferQueue)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.awaitMatch(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.xfer(java.base@11.0.22/Unknown Source)
at java.util.concurrent.LinkedTransferQueue.take(java.base@11.0.22/Unknown Source)
at org.opensearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:176)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.22/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.22/Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.22/Unknown Source)
at java.lang.Thread.run(java.base@11.0.22/Unknown Source)