Opensearch process is failed to start. Tried running it as a systemd service, as well as directly from /bin/opensearch.
As found in logs, it’s failing due to “unable to load JNA native support library, native methods will be disabled. java.lang.UnsatisfiedLinkError: /tmp/opensearchxxxxxx” error.
NOTE: I am installing it in Linux8 (VM) and /tmp is noexec for user “opensearch”. Hence I am getting this error and the process is failing to start. I got the reason why it’s failing to start but no luck how to fix it. I gone through various blogs/post relaed to similar issue in “Elasticsearch” and tried configs details below. But, nothing worked so far.
When I tried to fix it by setting “tmpdir” to some other locations for example: /opt/os_tmp/tmp OR /opt/opensearch/data/tmp, each time it crashes the JVM. If I leave default /tmp path, then process failed for error mentioned earlier (in heading).
Environment:
- Linux8 VM - installed Opensearch from Tar
- Unpackaged tar file on /opt and installed it there.
- Opensearch home dir: /opt/opensearch-1.2.4
- Created user “opensearch”. All files in opensearch home dir is owned by user “opensearch”.
- Custom tmpdir locations are also owned by user “opensearch”.
- Using default data and log folders (no custom path). So, data path is: /opt/opensearch-1.2.4/data and logs path is “/opt/opensearch-1.2.4/log”
Therefore - there are two scenarios here -
Scenario#1 - When there is no custom “tmpdir” setting (just default settings for tmpdir), then opensearch throw error saying “unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkErrorxxxxxx”
Scenario#2 - When I change “tmpdir” path to other location using either of the method given below, JVM get crashs.
2.1. Passed tmpdir path as an argument to commandline while running opensearch. But, no luck, JVM crashed
sample command:
[opensearch@osmaster1 opensearch-1.2.4]$ OPENSEARCH_JAVA_OPTS=“$OPENSEARCH_JAVA_OPTS -Djava.io.tmpdir=/opt/os_tmp/tmp” ./bin/opensearch
[opensearch@osmaster1 testadmin]$ ls -ltr /opt
drwxr-xr-x. 3 opensearch opensearch 17 Mar 31 18:55 os_tmp
drwxr-xr-x. 11 opensearch opensearch 275 Mar 31 19:39 opensearch-1.2.4
[opensearch@osmaster1 testadmin]$ ls -ltr /opt/os_tmp
drwxrwxrwx. 2 opensearch opensearch 6 Apr 1 13:27 tmp
2.2. Overridden tmpdir path in “/opt/opensearch-1.2.4/config/jvm.options”, again no luck, JVM crashed.
Default path in jvm.options file: -Djava.io.tmpdir=${OPENSEARCH_TMPDIR}"
Tried changing to : -Djava.io.tmpdir=data/tmp
and -Djava.io.tmpdir=/opt/os_tmp/tmp
JVM crashed for both path I tried above
2.3. Added tmpdir path in “/opt/opensearch-1.2.4/bin/opensearch-env” file, but again, JVM crashed.
Tried below settings separately -
#OPENSEARCH_JAVA_OPTS=“$OPENSEARCH_JAVA_OPTS -Djava.io.tmpdir=/opt/opensearch-1.2.4/data/tmp”
#OPENSEARCH_TMPDIR=“/opt/os_tmp/tmp”
#TMPDIR=“/opt/os_tmp/tmp”
#OPENSEARCH_TMPDIR=“$OPENSEARCH_HOME/data/tmp”
#TMPDIR=“$OPENSEARCH_HOME/data/tmp”
Below is the Error stack trace when using default settings for tmpdir (/tmp)
[opensearch@osmaster1 opensearch-1.2.4]$ ./bin/opensearch
[2022-04-01T13:00:23,035][WARN ][o.o.b.Natives ] [osmaster1] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/opensearch-1073614746573548365/jna9916496385695771284.tmp: /tmp/opensearch-1073614746573548365/jna9916496385695771284.tmp: failed to map segment from shared object
at jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?]
at jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:383) ~[?:?]
at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:227) ~[?:?]
at jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:169) ~[?:?]
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2407) ~[?:?]
at java.lang.Runtime.load0(Runtime.java:747) ~[?:?]
at java.lang.System.load(System.java:1857) ~[?:?]
at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1018) ~[jna-5.5.0.jar:5.5.0 (b0)]
at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:988) ~[jna-5.5.0.jar:5.5.0 (b0)]
at com.sun.jna.Native.<clinit>(Native.java:195) ~[jna-5.5.0.jar:5.5.0 (b0)]
at java.lang.Class.forName0(Native Method) ~[?:?]
at java.lang.Class.forName(Class.java:377) ~[?:?]
at org.opensearch.bootstrap.Natives.<clinit>(Natives.java:58) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:123) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-1.2.4.jar:1.2.4]
at org.opensearch.cli.Command.main(Command.java:101) [opensearch-cli-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135) [opensearch-1.2.4.jar:1.2.4]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101) [opensearch-1.2.4.jar:1.2.4]
[2022-04-01T13:00:23,045][WARN ][o.o.b.Natives ] [osmaster1] cannot check if running as root because JNA is not available
[2022-04-01T13:00:23,046][WARN ][o.o.b.Natives ] [osmaster1] cannot install system call filter because JNA is not available
[2022-04-01T13:00:23,046][WARN ][o.o.b.Natives ] [osmaster1] cannot register console handler because JNA is not available
[2022-04-01T13:00:23,048][WARN ][o.o.b.Natives ] [osmaster1] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2022-04-01T13:00:23,048][WARN ][o.o.b.Natives ] [osmaster1] cannot getrlimit RLIMIT_AS because JNA is not available
[2022-04-01T13:00:23,048][WARN ][o.o.b.Natives ] [osmaster1] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2022-04-01T13:00:23,264][INFO ][o.o.n.Node ] [osmaster1] version[1.2.4], pid[164586], build[tar/e505b10357c03ae8d26d675172402f2f2144ef0f/2022-01-14T03:38:06.881862Z], OS[Linux/4.18.0-348.12.2.el8_5.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]
[2022-04-01T13:00:23,265][INFO ][o.o.n.Node ] [osmaster1] JVM home [/opt/opensearch-1.2.4/jdk], using bundled JDK [true]
[2022-04-01T13:00:23,265][INFO ][o.o.n.Node ] [osmaster1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-1073614746573548365, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/opt/opensearch-1.2.4/plugins/opensearch-performance-analyzer/pa_config/opensearch_security.policy, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/opt/opensearch-1.2.4, -Dopensearch.path.conf=/opt/opensearch-1.2.4/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2022-04-01T13:00:24,310][INFO ][o.o.s.s.t.SSLConfig ] [osmaster1] SSL dual mode is disabled
[2022-04-01T13:00:24,311][INFO ][o.o.s.OpenSearchSecurityPlugin] [osmaster1] OpenSearch Config path is /opt/opensearch-1.2.4/config
[2022-04-01T13:00:24,598][INFO ][o.o.s.s.DefaultSecurityKeyStore] [osmaster1] JVM supports TLSv1.3
[2022-04-01T13:00:24,600][INFO ][o.o.s.s.DefaultSecurityKeyStore] [osmaster1] Config directory is /opt/opensearch-1.2.4/config/, from there the key- and truststore files are resolved relatively
[2022-04-01T13:00:25,260][INFO ][o.o.s.s.DefaultSecurityKeyStore] [osmaster1] TLS Transport Client Provider : JDK
[2022-04-01T13:00:25,260][INFO ][o.o.s.s.DefaultSecurityKeyStore] [osmaster1] TLS Transport Server Provider : JDK
[2022-04-01T13:00:25,261][INFO ][o.o.s.s.DefaultSecurityKeyStore] [osmaster1] TLS HTTP Provider : JDK
[2022-04-01T13:00:25,261][INFO ][o.o.s.s.DefaultSecurityKeyStore] [osmaster1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2022-04-01T13:00:25,261][INFO ][o.o.s.s.DefaultSecurityKeyStore] [osmaster1] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2022-04-01T13:00:25,526][INFO ][o.o.s.OpenSearchSecurityPlugin] [osmaster1] Clustername: ing-os-tau
[2022-04-01T13:00:25,532][WARN ][o.o.s.OpenSearchSecurityPlugin] [osmaster1] Directory /opt/opensearch-1.2.4/config has insecure file permissions (should be 0700)
[2022-04-01T13:00:25,678][INFO ][o.o.p.c.PluginSettings ] [osmaster1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2022-04-01T13:00:26,380][INFO ][o.o.i.r.ReindexPlugin ] [osmaster1] ReindexPlugin reloadSPI called
[2022-04-01T13:00:26,382][INFO ][o.o.i.r.ReindexPlugin ] [osmaster1] Unable to find any implementation for RemoteReindexExtension
[2022-04-01T13:00:26,398][INFO ][o.o.j.JobSchedulerPlugin ] [osmaster1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2022-04-01T13:00:26,402][INFO ][o.o.j.JobSchedulerPlugin ] [osmaster1] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2022-04-01T13:00:26,426][INFO ][o.o.j.JobSchedulerPlugin ] [osmaster1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2022-04-01T13:00:26,428][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [aggs-matrix-stats]
[2022-04-01T13:00:26,428][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [analysis-common]
[2022-04-01T13:00:26,429][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [geo]
[2022-04-01T13:00:26,429][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [ingest-common]
[2022-04-01T13:00:26,429][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [ingest-geoip]
[2022-04-01T13:00:26,429][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [ingest-user-agent]
[2022-04-01T13:00:26,429][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [lang-expression]
[2022-04-01T13:00:26,429][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [lang-mustache]
[2022-04-01T13:00:26,431][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [lang-painless]
[2022-04-01T13:00:26,432][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [mapper-extras]
[2022-04-01T13:00:26,432][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [opensearch-dashboards]
[2022-04-01T13:00:26,432][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [parent-join]
[2022-04-01T13:00:26,432][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [percolator]
[2022-04-01T13:00:26,432][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [rank-eval]
[2022-04-01T13:00:26,432][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [reindex]
[2022-04-01T13:00:26,433][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [repository-url]
[2022-04-01T13:00:26,433][INFO ][o.o.p.PluginsService ] [osmaster1] loaded module [transport-netty4]
[2022-04-01T13:00:26,433][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-alerting]
[2022-04-01T13:00:26,433][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-anomaly-detection]
[2022-04-01T13:00:26,434][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-asynchronous-search]
[2022-04-01T13:00:26,434][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-cross-cluster-replication]
[2022-04-01T13:00:26,434][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-index-management]
[2022-04-01T13:00:26,434][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-job-scheduler]
[2022-04-01T13:00:26,434][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-knn]
[2022-04-01T13:00:26,434][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-observability]
[2022-04-01T13:00:26,434][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-performance-analyzer]
[2022-04-01T13:00:26,435][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-reports-scheduler]
[2022-04-01T13:00:26,435][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-security]
[2022-04-01T13:00:26,435][INFO ][o.o.p.PluginsService ] [osmaster1] loaded plugin [opensearch-sql]
[2022-04-01T13:00:26,452][INFO ][o.o.s.OpenSearchSecurityPlugin] [osmaster1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2022-04-01T13:00:26,473][INFO ][o.o.e.NodeEnvironment ] [osmaster1] using [1] data paths, mounts [[/opt (/dev/mapper/rootvg-optvol)]], net usable_space [4.1gb], net total_space [4.9gb], types [xfs]
[2022-04-01T13:00:26,474][INFO ][o.o.e.NodeEnvironment ] [osmaster1] heap size [1gb], compressed ordinary object pointers [true]
[2022-04-01T13:00:26,548][INFO ][o.o.n.Node ] [osmaster1] node name [osmaster1], node ID [uLpIZav7Tb-LhXX1ryh1Qw], cluster name [ing-os-tau], roles [master, remote_cluster_client, ingest]
[2022-04-01T13:00:29,777][WARN ][o.o.s.c.Salt ] [osmaster1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2022-04-01T13:00:29,797][INFO ][o.o.s.a.i.AuditLogImpl ] [osmaster1] Message routing enabled: true
[2022-04-01T13:00:29,843][INFO ][o.o.s.f.SecurityFilter ] [osmaster1] <NONE> indices are made immutable.
[2022-04-01T13:00:30,088][INFO ][o.o.a.b.ADCircuitBreakerService] [osmaster1] Registered memory breaker.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/opt/opensearch-1.2.4/plugins/opensearch-anomaly-detection/protostuff-runtime-1.7.4.jar) to field java.lang.Throwable.cause
WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2022-04-01T13:00:30,683][INFO ][o.o.t.NettyAllocator ] [osmaster1] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=1gb}]
[2022-04-01T13:00:30,780][INFO ][o.o.d.DiscoveryModule ] [osmaster1] using discovery type [zen] and seed hosts providers [settings]
[2022-04-01T13:00:31,197][WARN ][o.o.g.DanglingIndicesState] [osmaster1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2022-04-01T13:00:31,506][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [osmaster1] PerformanceAnalyzer Enabled: false
[2022-04-01T13:00:31,557][INFO ][o.o.n.Node ] [osmaster1] initialized
[2022-04-01T13:00:31,557][INFO ][o.o.n.Node ] [osmaster1] starting ...
[2022-04-01T13:00:31,648][INFO ][o.o.t.TransportService ] [osmaster1] publish_address {10.56.128.9:9300}, bound_addresses {10.56.128.9:9300}
[2022-04-01T13:00:31,930][INFO ][o.o.b.BootstrapChecks ] [osmaster1] bound or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
ERROR: OpenSearch did not exit normally - check the logs at /opt/opensearch-1.2.4/logs/ing-os-tau.log
[2022-04-01T13:00:31,950][INFO ][o.o.n.Node ] [osmaster1] stopping ...
[2022-04-01T13:00:31,952][INFO ][o.o.s.a.r.AuditMessageRouter] [osmaster1] Closing AuditMessageRouter
[2022-04-01T13:00:31,953][INFO ][o.o.s.a.s.SinkProvider ] [osmaster1] Closing InternalOpenSearchSink
[2022-04-01T13:00:31,953][INFO ][o.o.s.a.s.SinkProvider ] [osmaster1] Closing DebugSink
[2022-04-01T13:00:31,970][INFO ][o.o.n.Node ] [osmaster1] stopped
[2022-04-01T13:00:31,971][INFO ][o.o.n.Node ] [osmaster1] closing ...
[2022-04-01T13:00:31,977][INFO ][o.o.s.a.i.AuditLogImpl ] [osmaster1] Closing AuditLogImpl
[2022-04-01T13:00:31,981][INFO ][o.o.n.Node ] [osmaster1] closed
[opensearch@osmaster1 opensearch-1.2.4]$ ^C
Process failed logs when JVM Crashes:
[opensearch@osmaster1 opensearch-1.2.4]$ OPENSEARCH_JAVA_OPTS="$OPENSEARCH_JAVA_OPTS -Djava.io.tmpdir=/opt/os_tmp/tmp" ./bin/opensearch
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007f8d48da5c95, pid=165097, tid=165266
#
# JRE version: OpenJDK Runtime Environment AdoptOpenJDK (15.0.1+9) (build 15.0.1+9)
# Java VM: OpenJDK 64-Bit Server VM AdoptOpenJDK (15.0.1+9, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [jna11005614729983069530.tmp+0x12c95] ffi_prep_closure_loc+0x15
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h %e" (or dumping to /opt/opensearch-1.2.4/core.165097)
#
# An error report file with more information is saved as:
# logs/hs_err_pid165097.log
#
# If you would like to submit a bug report, please visit:
# https://github.com/AdoptOpenJDK/openjdk-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
Aborted (core dumped)
[opensearch@osmaster1 opensearch-1.2.4]$