OpenSearch 2.15.0 increased jvm heap usage

[smbambling]

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch Version : 2.15.0

Describe the issue:
Higher Heap Usage on WARM data nodes after upgrading from 2.14.0 to 2.15.0. Master and HOT data nodes see a slight bump in HEAP usage.

GC is still running on what looks like the same interval as before the upgrade and completing just as quickly as before ( see screen shots )

Configuration:
Our cluster has been configured using the OpenSearch Helm chart which was bumpped from 2.20 to 2.21 for the upgrade.

Each data node (pod) both WARM and HOT are configured with 32G of memory and 16G of HEAP. No memory values were changed during the upgrade and no additional services have been added to the cluster for increased volume of log entries ingested.

Relevant Logs or Screenshots:

JVM Heap Usage / Percent before and after upgrade

The only notable difference I can sus out easily is the fact that now none of my WARM nodes have any query cache. I’m unsure if this is related, I assume some query cache is healthier for the cluster.
– This could also be a bug in the Prometheus exporter plugin as well, but no GH issues have been reported and 2.15.0 is a supported cluster version.

[smbambling]

I was limited to only putting one media item ( screenshot ) as a new user so continuing adding some as a replys

Garbage Collection Graphs

[smbambling]

Query Cache Size Graphs

[reta]

Hi @smbambling , the problem is very likely related to Jackson regression (see please [1]) that should be fixed in 2.16.0 (planned release date is August 6th). Thank you.

UPD: apologies for misleading issue, the Jackson issue impacts 2.14.x and since you are using 2.15.0, it should not be there. The absence of query cache may cause higher GC activity, any particular reasons why it this configuration was changed for WARM nodes?

[1] [BUG] Huge JVM Heap Usage · Issue #13927 · opensearch-project/OpenSearch · GitHub

[smbambling]

No configuration changes were made during the upgrade process to any of the nodes (master,hot or warm). Over the last couple of day the heap foot print has changes and reduced. However its still much different then before the upgrade to 2.15.0 and has large spikes of heap usage that rise above an 85% threshold.

[smbambling]

There is some query caching that is being performed on the warm nodes however it also is much different from before the upgrade