OpenSearch fails to start using alternate Java tmp directory

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

OpenSearch 2.12
RHEL 8.9

Describe the issue:

OpenSearch will not start when using alternate location for java temp directory. Our security posture requires /tmp to have the noexec attribute set. That setting prevents OpenSearch from starting. IF I have /tmp mounted w/o noexec flag, OpenSearch will start.

I’ve set an alternate java temp location in jvm.options with:
-Djava.io.tmpdir=/usr/share/opensearch/tmp

Confirmed that directory is writable.

Configuration:

SELinux: Permissive
fapolicyd: disabled

Relevant Logs or Screenshots:

Opensearch log:

[2024-02-28T15:32:24,354][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [MNOCE-OPENSEARCH-01] fatal error in thread [main], exiting
java.lang.ExceptionInInitializerError: null
at org.opensearch.bootstrap.JNANatives.definitelyRunningAsRoot(JNANatives.java:192) ~[opensearch-2.12.0.jar:2.12.0]
at org.opensearch.bootstrap.Natives.definitelyRunningAsRoot(Natives.java:83) ~[opensearch-2.12.0.jar:2.12.0]
at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:123) ~[opensearch-2.12.0.jar:2.12.0]
at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191) ~[opensearch-2.12.0.jar:2.12.0]
at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404) ~[opensearch-2.12.0.jar:2.12.0]
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181) ~[opensearch-2.12.0.jar:2.12.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172) ~[opensearch-2.12.0.jar:2.12.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.12.0.jar:2.12.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.12.0.jar:2.12.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.12.0.jar:2.12.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138) ~[opensearch-2.12.0.jar:2.12.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104) ~[opensearch-2.12.0.jar:2.12.0]
Caused by: java.lang.UnsupportedOperationException: Failed to allocate closure
at com.sun.jna.Native.registerMethod(Native Method) ~[jna-5.13.0.jar:5.13.0 (b0)]
at com.sun.jna.Native.register(Native.java:1906) ~[jna-5.13.0.jar:5.13.0 (b0)]
at com.sun.jna.Native.register(Native.java:1775) ~[jna-5.13.0.jar:5.13.0 (b0)]
at com.sun.jna.Native.register(Native.java:1493) ~[jna-5.13.0.jar:5.13.0 (b0)]
at org.opensearch.bootstrap.JNACLibrary.(JNACLibrary.java:64) ~[opensearch-2.12.0.jar:2.12.0]
… 12 more

2

Hi @tomusn83,

Have you seen the below (I stumbled on it and it looks related, but might not be):
Opensearch process FAILED to start due to "java.lang.UnsatisfiedLinkError: /tmp/opensearch-***** failed to map segment from shared object" error

Let me know if it helps, if not I can try to reproduce it in my lab and look at it a bit closer.

Best,
mj

3

Mantas,

Thanks! That did resolve the issue.

Solution:
Changing the home dir for the opensearch user in /etc/passwd to the install path (/usr/share/opensearch) allowed opensearch to start.

Possible reason:
I “think” I see why this is happening. In /etc/opensearch/opensearch-performance-analyzer/log4j2.xml file there are some xml attributes as follows:

fileName=“${sys:opensearch.path.home:-/tmp}/logs/PerformanceAnalyzer.log”

As the opensearch user home path in /etc/passwd was invalid, the performance analyzer was writing to /tmp. Not sure why the noexec attribute on /tmp broke Java as all these entries appear to just be writing to logs, but that would take a deeper dive.

2 Likes
4

@tomusn83, thanks for sharing your solution summary!