How to reduce the node's heap size?

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.7.0

Describe the issue:

image
image875×169 31.9 KB

almost nodes full heap size
how to reduce the heap size??

2

I don’t think the reduction will help. I think you have exhausted your resources and need to increase the number of nodes.

3

It also depends on what uses heap. Could be garbage that is collected later on, though it’s unlikely to see high usage on all data nodes like that. You’ll want to monitor your OpenSearch cluster to get more information about what’s using heap. I’m biased, but we have a nice monitoring tool that you can use - you’ll see both the JVM-level stuff (i.e. is it garbage?) and the OpenSearch-level stuff (e.g. how much memory segments use? what about caches?).

If you just want some quick ideas, here are a few from my colleague: https://www.youtube.com/watch?v=3ilTrM7tTWs

On 5 nodes with 31GB of heap, unless you either have 10-20TB data on them or you run very expensive queries (or both), I don’t see why you’d run out of heap. Maybe it really is garbage and the GCs are letting them all slowly grow before cleaning them up.

4

IIRC it allocates various buffers as a ratio of the given heap. So, these ninety-something might be ok unless you evidence high gc spikes.
I agree with Radu regarding reducing heaps.

5

I solved it. The reason why had been increased heap size is a lot of shards. I deleted replica and reduce a number of shards. As result,

image
image898×166 30.5 KB

but heap size imbalance occurred in all nodes.