Increase Opensearch node heap size

OpenSearch
, , ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch: 2.11.1
Dashboard: 2.11.1

Describe the issue:
I have used opensearch project helm to install opensarch in kubernetes env (GitHub - opensearch-project/helm-charts: ☸ A community repository for Helm Charts of OpenSearch Project.). I have been trying to update the default heap size, dut did not had luck.
So far I tried

  1. opensearchJavaOpts: “-Xmx3g -Xms3g” in values.yml → did not work

  2. tried to update jvm.options → I dont have update permission so no chance to update it, even though I will change the parameter, on container restart, it will again reset with default, so surely this will be not the correct way.

Do you know, what would be correct way to increase the heapsize of node using helm chart

Configuration:

Relevant Logs or Screenshots:

] [opensearch-cluster-master-0] JVM home [/usr/share/opensearch/jdk], using bundled JDK/JRE [true]
[2024-02-18T07:32:55,189][INFO ][o.o.n.Node ] [opensearch-cluster-master-0] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-11646835306537532458, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=/usr/share/opensearch/config/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms4g, -Xmx4g, -XX:MaxDirectMemorySize=2147483648, -Dopensearch.path.home=/usr/share/opensearch, -Dopensearch.path.conf=/usr/share/opensearch/config, -Dopensearch.distribution.type=tar, -Dopensearch.bundled_jdk=true]
[2024-02-18T07:32:56,678][INFO ][o.o.s.s.t.SSLConfig ] [opensearch-cluster-master-0] SSL dual mode is disabled

2

Hey @kiranpophale ,

Sadly setting OPENSEARCH_JAVA_OPTS would not help since you have now these settings present twice in the command like:

...  -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25 ... --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -Dopensearch.cgroups.hierarchy.override=/, -Xms4g, -Xmx4g, -

I believe the jvm.options would be the only reliable way to go at the moment. Thank you.

3

@kiranpophale

This is the proper way and it worked for me.

The OpenSearch pod uses this option as an environmental variable in the StatefulSet.

./charts/opensearch/templates/statefulset.yaml

        - name: OPENSEARCH_JAVA_OPTS
          value: "{{ .Values.opensearchJavaOpts }}"

After running helm upgrade <helm_charts_name> <path_to_values.yml>, the statefulset pod should have this option already updated. As part of the helm upgrade, the OpenSearch pods will be terminated and created with new JAVA values. You’ll need to give the pod a moment to start and apply new values.

4

thank you for your reply.
I tried to perform the steps you mentioned but no luck to me,

What I see in values.ymal file inside the opensearch charts, it has been set to 512M
opensearchJavaOpts: “-Xmx512M -Xms512M”
however it does set the 1g default heap size, I tried to expore it but I did not find any clue here

I did check in dashboard with dev tools and observed below ouput

“mem”: {
“heap_init_in_bytes”: 536870912,
“heap_max_in_bytes”: 536870912,
“non_heap_init_in_bytes”: 7667712,
“non_heap_max_in_bytes”: 0,
“direct_max_in_bytes”: 0
},
As per this it looks like it does allocate 50% memory to opensearch node, correct me if I did understand wrongly

Do you know any other alternative or have you checked with opensearch chart 2.11.1 ?

5

thank you for your reply, since we uses K8S cluster to spin up the open search pods/node, it seems I dont have require permission to modify the jvm.options file. It would be problem in case container restart, it will again set to original value.

6

@kiranpophale How much RAM did you assign to the k8s node?

7

I did verify and it is 12 GB

8

Could you share the command that produced this output?

9

@pablo , I have used the dev tool option from opensearch dashboard, Please see below screen shot,

10

@kiranpophale Thanks, I’ve just found it. I’ve used curl instead.

curl --insecure -u admin:admin https://localhost:9200/_nodes/jvm?pretty

This is the result.

        "mem" : {
          "heap_init_in_bytes" : 1073741824,
          "heap_max_in_bytes" : 1073741824,
11

@kiranpophale Could you run the following commands in your K8s cluster and share the outputs?

kubectl exec -it <opensearch_pod> -- env|grep OPENSEARCH_JAVA_OPTS

kubectl get statefulset <opensearch_statefulset> -o yaml|grep OPENSEARCH_JAVA_OPTS -A1
12

I have verified and found it was 512M default

Now totally confused desipte I have set the values explicitly, it was not applying, actually I can see my other configuration like self certificate config works fine in my values.yaml

13

@kiranpophale How exactly did you update the charts?
Did you run helm upgrade after applying new settings in values.yml? Please share your command.

It looks like your statefulset wasn’t updated.

You have two choices here.

  1. Update deployment with helm upgrade and updated values.yml file.
  2. Use the below command and update the statefulset directly
kubectl edit statefulset <Opensearch_statefulset>

After following one of the steps, the existing OpenSearch pods will be terminated and new pods with settings created.

14

@pablo, thanks for all support, It was my silly mistake, I had configured opensearchJavaOpts at wrong place, it should be under the opensearch: in values.yaml, It is working now.