Opensearch ignoring "-XX:-UsePerfData" to disable "perfdata" java feature (to avoid creating "hsperfdata_opensearch" dir in /tmp)

OpenSearch
1

Hi members,

Looking forward for some help in issues we’re facing in setting up our Opensearch environement (Linux VM - using opensearch tar ball)

Issue:
Opensearch always create “hsperfdata_{userid}” dir in “/tmp” folder. In our environment (Linux8 VM), “/tmp” is “noexec”. We cannot remove noexec behaviour due to security concerns.

Initially, we tried to fix it by changing “hsperfdata_opensearch” location from “/tmp” to some other custom location e.g. /opt/os/tmp but nothing worked. Passed below listed variables/arguments in jvm.options / JAVA_OPTS but no luck, nothing worked. Opensearch continue to create “hsperfdata_opensearch” dir in /tmp folder only.

-Djava.io.tmpdir=/opt/os/tmp
TMPDIR=/opt/os/tmp
TMP=/opt/os/tmp
-Djna.tmpdir=/opt/os/tmp

Eventually, we thought to disable this “hsperfdata” feature in our opensearch environment. “hsperfdata” is a java feature and default location in java for this is “/tmp” (hardcoded).

To disable it, (as Java documentaion suggested), we passed java option “-XX:-UsePerfData” but opensearch is keep on ignoring it. It’s continue to create “hsperfdata_opensearch” in /tmp.

We set “-XX:-UsePerfData” in “OPENSEARCH_HOME/config/jvm.options” as well as tried in JAVA command in “OPENSEARCH_HOME/bin/opensearch” executable file (hard coded for testing purpose) but no luck. Opensearch is not at all entertaining this Java option “-XX:-UsePerfData”. We even stripped out triggering command for opensearch, passed “-XX:-UsePerfData” in that java command to run opensearch, but it’s still creating “hsperfdata_opensearch” in /tmp dir only.

NOTE: To test the behaviour independently, we created a test Hello World Java program. Tried Java option “-XX:-UsePerfData” there, and it shows expected behaviour, as follows:

  • Ran Java “HelloWorld” program WITHOUT “-XX:-UsePerfData” - a directory “hsperfdata_opensearch” got created in /tmp

sample command:
/opt/opensearch-1.2.4/jdk/bin/java -Djava.io.tmpdir=/opt/os/tmp HelloWorld

  • Ran Java “HelloWorld” program WITH “-XX:-UsePerfData” - No “hsperfdata_opensearch” got created in /tmp

sample command:
/opt/opensearch-1.2.4/jdk/bin/java -XX:-UsePerfData -Djava.io.tmpdir=/opt/os/tmp HelloWorld

However, in case of Opensearch, irrespective to if we pass Java option “-XX:-UsePerfData” or not, it always create “hsperfdata_opensearch” in “/tmp” dir.
NOTE: In “hsperfdata_opensearch” → opensearch is the user we use to run the opensearch process.

Environment:
Linux8, VM
Opensearch1.4 which uses JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/15.0.1/15.0.1+9]

2

Opensearch is actually ignoring all my custom configurations done in “jvm.options” or in OPENSEARCH_JAVA_OPTS etc… I want to change /tmp location to some custom location “/opt/os/tmp” to fix issue (Error below). But nothing is working. Could any of you please suggest how to fix it in Opensearch?

[2022-04-01T10:18:09,587][WARN ][o.o.b.Natives            ] [npsy7sretools01] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/opensearch-1131263490975920622/jna1578977023810974048.tmp: /tmp/opensearch-1131263490975920622/jna1578977023810974048.tmp: failed to map segment from shared object
3

Hey Shadab, this actually might be better listed as a bug in the OpenSearch project. Here is the link to create one:

https://github.com/opensearch-project/OpenSearch/issues/new?assignees=&labels=bug%2C+untriaged&template=bug_template.md&title=[BUG]

This is the first I am seeing a security concern about using tmp in particular when a whole new VM is being used to host it. Do you have a link to something I can read to learn a bit more about why this is a concern? Thank you, and looking forward to looking into it more!

4

Hi @dtaivpp

I got the root cause of the issue which made us thinking to try default perf location (/tmp) to some other location where user “opensearch” has permissiones. Therefore, we no longer required to change perf location from /tmp to others.

Below are more details:

Trigger point of /tmp related location issues is – “when we run opensearch service on Linux as a user who doesn’t have home dir on executable path”. We start getting JNA error unsatisfied link related.

[2022-04-01T13:00:23,035][WARN ][o.o.b.Natives            ] [osmaster1] unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: /tmp/opensearch-1073614746573548365/jna9916496385695771284.tmp: /tmp/opensearch-1073614746573548365/jna9916496385695771284.tmp: failed to map segment from shared object
        at jdk.internal.loader.NativeLibraries.load(Native Method) ~[?:?]

So, when user doesnt have home dir, opensearch try to use “/tmp” dir for JNA and other stuff. But, since user “opensearch” doesn’t have execute permissions, it throws “unsatisfied link” error.

To try fixing it - we tried io.dir from /tmp to other other location but in that case JVM was crashing.

How to fix it:

Step#1 - Create user e.g. “opensearch” with a home dir on executable path for this user.
For example: Home dir “/opt/opensearch” for user “opensearch” .

Sample command: useradd -r -m -d /opt/opensearch opensearch

Step#2 - install opensearch in this dir /opt/opensearch/
So, your installation will look like /opt/opensearch/opensearch1.4.xx

More details about JNA / Unsatisfied link error I was facing:

Hope this help others facing similar issue.

2 Likes
5

Thank you for coming back and sharing your findings with the community! This is a great write-up!