Exhausted SWAP during the build

Trying to run build OpenSearch from upstream on Fedora and I repeatedly run into SWAP file exhaustion and the build process is killed by the system.

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

A fresh content from upstream.

$ git log -n 1 --oneline
e44b3f1993d (HEAD -> main, origin/main, origin/HEAD) [Remote Store] Clear feature flag set during unit test runs (#7523)
$ uname -a
Linux fedora 6.2.7-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Mar 17 16:16:00 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ java --version
openjdk 11.0.18 2023-01-17
OpenJDK Runtime Environment (Red_Hat-11.0.18.0.10-1.fc37) (build 11.0.18+10)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.18.0.10-1.fc37) (build 11.0.18+10, mixed mode, sharing)

Describe the issue:

$ .gradlew clean build --continue

After ~10 minutes the swap file gets filled quite quickly from 1G to 4G and about 1-2 minutes later it grows from 4G to 8G. At this point the system kills the build.

The following is a screenshot of gradle output around the 8G swap size. Perhaps one (or more) of these tests is the root cause?

As you can see the swap is completely full.

Anyone has experience with this? Shall I try to expand the swap space or use machine with more RAM?

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            27Gi       1.4Gi        24Gi       0.0Ki       1.2Gi        25Gi
Swap:          8.0Gi       1.1Gi       6.9Gi

Regards,
Lukáš

Answering my own question: To fix that problem it was necessary to find a sweetspot for HW resources available on my machine.

  1. I lowered down the -Xmx3g to -Xmx2g in gradle.properties file (in the root). This helped a bit but still the swap was eventually exhausted and the build process was terminated by the system.
  2. I started limiting the number of parallel jobs the gradle was running. I ended up with: gradlew clean build --max-workers=6. This way the swap was not used at all (I mean no large tasks were moved into it, the only exception was the :qa module) yet the system memory was utilized pretty well most of the time.

Hint:

I found the command smem very useful in this context: watch smem -p -s swap -r

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.