Enabling performance analyzer kills ES node

pemontto · April 9, 2019, 10:45am

OS: CentOS Linux release 7.6.1810 (Core)
OpenDistro: 0.8.0-1 installed from repo

When I have performance analyzer installed my ES node refuses to start, giving me the below error. Starts fine when I yum remove opendistro-performance-analyzer

[2019-04-09T20:42:22,196][ERROR][c.a.o.e.p.o.OSGlobals    ] [dev-es-b-3] Error in reading/parsing clk.tck value: java.lang.NumberFormatException: null
java.lang.NumberFormatException: null
	at java.lang.Long.parseUnsignedLong(Long.java:680) ~[?:1.8.0_152]
	at java.lang.Long.parseUnsignedLong(Long.java:746) ~[?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.OSGlobals.getScClkTckFromConfig(OSGlobals.java:65) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.OSGlobals.<clinit>(OSGlobals.java:43) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.ThreadCPU.<init>(ThreadCPU.java:139) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.ThreadCPU.<clinit>(ThreadCPU.java:31) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics_generator.linux.LinuxOSMetricsGenerator.getPagingActivityGenerator(LinuxOSMetricsGenerator.java:56) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.OSMetricsCollector.collectMetrics(OSMetricsCollector.java:50) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.lambda$run$0(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:103) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:101) [opendistro_performance_analyzer-0.8.0.0.jar:0.8.0.0]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.run(PerformanceAnalyzerMetricsCollector.java:57) [opendistro_performance_analyzer-0.8.0.0.jar:0.8.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
[2019-04-09T20:42:22,271][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [dev-es-b-3] fatal error in thread [pool-2-thread-2], exiting
java.lang.NoClassDefFoundError: com/sun/tools/attach/VirtualMachine
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.runAttachDump(ThreadList.java:129) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.lambda$runThreadDump$3(ThreadList.java:201) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:103) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:101) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.runThreadDump(ThreadList.java:201) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.getNativeTidMap(ThreadList.java:96) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.OSMetricsCollector.collectMetrics(OSMetricsCollector.java:56) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.lambda$run$0(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:103) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:101) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.run(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: java.lang.ClassNotFoundException: com.sun.tools.attach.VirtualMachine
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_152]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_152]
	at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814) ~[?:1.8.0_152]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_152]
	... 16 more

kdinta · April 10, 2019, 5:52pm

Thanks for reporting the issue. Can you please let us know, how you are starting ES with plugin: are you using opendistro RPM or docker image or you built plugin & used plugin install script?

If it is RPM or docker, we have taken care of the following.

But if you are building plugin & installed, the following things required to set:

pass clk.tck= as -D option for java (for linux: can get the value by executing: /usr/bin/getconf CLK_TCK)
If using JDK 8, need to bring your tools.jar (from JDK), and have it in the class path (as specified in Troubleshoot - Open Distro Documentation); If using >= JDK 9, need to set: jdk.attach.allowAttachSelf=true as -D option
ES security policy need to set to: java.security.policy=<ES_HOME>/plugins/opendistro_performance_analyzer/pa_config/es_security.policy as -D option

Will be adding an issue (on github) for more detailed error messages if these are not set properly

pemontto · April 14, 2019, 11:58pm

Thanks! I was installing with the RPM though I was deploying with an existing ES config template but must have missed these reqs.

kdinta · April 15, 2019, 3:50pm

Thanks for more details.

For JDK 8 dependencies (tools.jar), even with RPM, it is bring your own tools.jar model; as mentioned in the documentation
Surprised to see the clk.tck exception, as we made sure to set it with the RPM. Which OS you are trying?

pemontto · April 22, 2019, 1:49am

I’m on Centos7, I’m assuming it won’t be the RPMs problem it will be my ansible playbook using its own systemd config file. Thanks!

sendkb · August 7, 2019, 10:02pm

hi,
Were you able to get PeformanceAnalyzer to work in your environment. Let us know if you need more help

Topic		Replies	Views
How to enable/install performance analyzer on opensearch 3.0 in docker-compose Performance Analyzer	5	783	May 26, 2025
Elasticsearch crash when start, caused by performance analyzer plugin Performance Analyzer	0	1449	May 11, 2021
Performance Analyzer Webservice does not come up Performance Analyzer troubleshoot , configure	6	218	August 22, 2024
Performance Analyzer Failed to execute operation: Bad message Performance Analyzer	0	855	April 29, 2021
Performance analyzer not showing metrics Performance Analyzer	4	1456	December 20, 2019

Enabling performance analyzer kills ES node

Related topics