Enabling performance analyzer kills ES node

pemontto · April 9, 2019, 10:45am

OS: CentOS Linux release 7.6.1810 (Core)
OpenDistro: 0.8.0-1 installed from repo

When I have performance analyzer installed my ES node refuses to start, giving me the below error. Starts fine when I yum remove opendistro-performance-analyzer

[2019-04-09T20:42:22,196][ERROR][c.a.o.e.p.o.OSGlobals    ] [dev-es-b-3] Error in reading/parsing clk.tck value: java.lang.NumberFormatException: null
java.lang.NumberFormatException: null
	at java.lang.Long.parseUnsignedLong(Long.java:680) ~[?:1.8.0_152]
	at java.lang.Long.parseUnsignedLong(Long.java:746) ~[?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.OSGlobals.getScClkTckFromConfig(OSGlobals.java:65) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.OSGlobals.<clinit>(OSGlobals.java:43) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.ThreadCPU.<init>(ThreadCPU.java:139) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.os.ThreadCPU.<clinit>(ThreadCPU.java:31) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.metrics_generator.linux.LinuxOSMetricsGenerator.getPagingActivityGenerator(LinuxOSMetricsGenerator.java:56) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.OSMetricsCollector.collectMetrics(OSMetricsCollector.java:50) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.lambda$run$0(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:103) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:101) [opendistro_performance_analyzer-0.8.0.0.jar:0.8.0.0]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.run(PerformanceAnalyzerMetricsCollector.java:57) [opendistro_performance_analyzer-0.8.0.0.jar:0.8.0.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
[2019-04-09T20:42:22,271][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [dev-es-b-3] fatal error in thread [pool-2-thread-2], exiting
java.lang.NoClassDefFoundError: com/sun/tools/attach/VirtualMachine
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.runAttachDump(ThreadList.java:129) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.lambda$runThreadDump$3(ThreadList.java:201) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:103) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:101) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.runThreadDump(ThreadList.java:201) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.jvm.ThreadList.getNativeTidMap(ThreadList.java:96) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.OSMetricsCollector.collectMetrics(OSMetricsCollector.java:56) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.lambda$run$0(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.lambda$invokePrivileged$1(PerformanceAnalyzerPlugin.java:103) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerPlugin.invokePrivileged(PerformanceAnalyzerPlugin.java:101) ~[?:?]
	at com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.PerformanceAnalyzerMetricsCollector.run(PerformanceAnalyzerMetricsCollector.java:57) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: java.lang.ClassNotFoundException: com.sun.tools.attach.VirtualMachine
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381) ~[?:1.8.0_152]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_152]
	at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814) ~[?:1.8.0_152]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_152]
	... 16 more

kdinta · April 10, 2019, 5:52pm

Thanks for reporting the issue. Can you please let us know, how you are starting ES with plugin: are you using opendistro RPM or docker image or you built plugin & used plugin install script?

If it is RPM or docker, we have taken care of the following.

But if you are building plugin & installed, the following things required to set:

pass clk.tck= as -D option for java (for linux: can get the value by executing: /usr/bin/getconf CLK_TCK)
If using JDK 8, need to bring your tools.jar (from JDK), and have it in the class path (as specified in Troubleshoot - Open Distro Documentation); If using >= JDK 9, need to set: jdk.attach.allowAttachSelf=true as -D option
ES security policy need to set to: java.security.policy=<ES_HOME>/plugins/opendistro_performance_analyzer/pa_config/es_security.policy as -D option

Will be adding an issue (on github) for more detailed error messages if these are not set properly

pemontto · April 14, 2019, 11:58pm

Thanks! I was installing with the RPM though I was deploying with an existing ES config template but must have missed these reqs.

kdinta · April 15, 2019, 3:50pm

Thanks for more details.

For JDK 8 dependencies (tools.jar), even with RPM, it is bring your own tools.jar model; as mentioned in the documentation
Surprised to see the clk.tck exception, as we made sure to set it with the RPM. Which OS you are trying?

pemontto · April 22, 2019, 1:49am

I’m on Centos7, I’m assuming it won’t be the RPMs problem it will be my ansible playbook using its own systemd config file. Thanks!

sendkb · August 7, 2019, 10:02pm

hi,
Were you able to get PeformanceAnalyzer to work in your environment. Let us know if you need more help

Topic		Replies	Views
Performance Analyzer plugin not parsing Performance Analyzer	1	2813	July 31, 2020
Performance analyzer not showing metrics Performance Analyzer	4	1314	December 20, 2019
Error writing entry 'NOT_INITIALIZED'. Cause /dev/shm/performanceanalyzer/xxxxx.tmp Performance Analyzer	2	1124	November 7, 2023
Removing performance analyzer plugin from opendistro docker Performance Analyzer configure	2	3761	May 12, 2022
Issues with 1.11 performance analyzer no data no bind variables have been provided Performance Analyzer	1	1226	February 7, 2023

Enabling performance analyzer kills ES node

Related Topics