Mem used in Bytes from Stats API is showing higher than Kubectl top pods response

[chirumanem]

Memory Used Bytes indicating by OpenSearch Stats API(/_nodes/stats/os?pretty) is higher than the Kubectl top pods command response.

The below is the response got from Stats (/_nodes/stats/os?pretty) API and it is indicating the below values
“mem” : {
“total_in_bytes” : 4294967296,
“free_in_bytes” : 32768,
“used_in_bytes” : 4294934528,
“free_percent” : 0,
“used_percent” : 100
},

But when we checked the used bytes using the Kubectl top command then we got the memory used bytes as 2535Mi

kubectl top pods --containers |grep data-1
eric-data-search-engine-data-1 data 688m 2535Mi
eric-data-search-engine-data-1 logshipper 5m 29Mi
eric-data-search-engine-data-1 metrics 4m 25Mi
eric-data-search-engine-data-1 tlsproxy 8m 20Mi

We would like to know why this difference in OpenSearch Stats API response. Could you please help us to understand this.

Note: There is no difference between Stats API and Kubectl top command response of memory used bytes when there is no data loaded into Opensearch.

[reta]

Hey @chirumanem,

Would be helpful to know which Opensearch version and JDK version you are using.
Thank you.

[chirumanem]

Hi @reta,
We are using Opensearch version 1.2.4 and JDK version 11.
Thank you

[reta]

Hi @chirumanem ,

Thanks, since it is still unclear which JDK 11 version you are on, I suspect you run into [1] (which should be fixed in 11.0.9). Thank you.

[1] https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8226575

[chirumanem]

Hi @reta ,
We are using JDK 11.0.15 version, but still having issues. Are there any other parameters we need to check.
Thank you

[radu.gheorghe]

Hello,

AFAIK, what you see in stats is what OpenSearch see as the OS-level used memory, while the top output is specifically the memory used by the OpenSearch process. I think that explains why you see much more reported by OpenSearch, because it sees other containers from the same node.

[chirumanem]

Hi @radu.gheorghe ,
Why does Opensearch needs to show OS-level used memory (including other process/containers in same node) ?

[radu.gheorghe]

Hi @chirumanem,

I wouldn’t say that it needs to, but it’s nice to know when you’re running out of system memory. The Stats API doesn’t assume you’re running in a container, in a bare-metal or VM deployment it maybe makes more sense. For a containerized environment, I guess you’ll get the same metrics from other sources.

[reta]

@chirumanem the only hypothesis I have left is that you are using CGroups v2, could you confirm that? The JVM support of CGroups v2 is coming [1] in JDK 11.0.16 release.

[1] [JDK-8230305] Cgroups v2: Container awareness - Java Bug System

[chirumanem]

@chirumanem

difference in OpenSearch Stats API response

Hi @radu.gheorghe ,
Hi,
As per the below output it is clear that data-0, data-1, data-5 & data-6 are not sharing the node with other pods/containers whereas data-2, data-3, data-4 & data-7 are sharing.

eric-data-search-engine-data-0 node-10-63-135-112
eric-data-search-engine-data-1 node-10-63-135-106
eric-data-search-engine-data-2 node-10-63-135-111
eric-data-search-engine-data-3 node-10-63-135-110
eric-data-search-engine-data-4 node-10-63-135-105
eric-data-search-engine-data-5 node-10-63-135-100
eric-data-search-engine-data-6 node-10-63-135-115
eric-data-search-engine-data-7 node-10-63-135-107
eric-data-search-engine-ingest-tls-67b64764b6-4ln6n node-10-63-135-103
eric-data-search-engine-ingest-tls-67b64764b6-bkt85 node-10-63-135-102
eric-data-search-engine-ingest-tls-67b64764b6-f6q9v node-10-63-135-109
eric-data-search-engine-ingest-tls-67b64764b6-gdfp2 node-10-63-135-114
eric-data-search-engine-ingest-tls-67b64764b6-pdm5t node-10-63-135-111
eric-data-search-engine-ingest-tls-67b64764b6-xq5cq node-10-63-135-101
eric-data-search-engine-ingest-tls-67b64764b6-zjpml node-10-63-135-105
eric-data-search-engine-ingest-tls-67b64764b6-zxdjv node-10-63-135-104
eric-data-search-engine-master-0 node-10-63-135-110
eric-data-search-engine-master-1 node-10-63-135-107
eric-data-search-engine-master-2 node-10-63-135-108

Example 1: The below is the response from the Stat APIs and kubectl top pods of the data-1, which does not contain other pods/containers in the same node:

Response from Stat APIs
eric-data-search-engine-data-1
“mem” : {
“total_in_bytes” : 4294967296,
“free_in_bytes” : 9699328,
“used_in_bytes” : 4285267968,
“free_percent” : 0,
“used_percent” : 100
},
Output from kubectl top pod
eric-data-search-engine-data-1 860m 2641Mi

Example 2: The below is the response from the kubectl top pods and Stat APIs of the data-3, which contains other pods/containers in the same node:

Response from Stat APIs
eric-data-search-engine-data-3
“mem” : {
“total_in_bytes” : 4294967296,
“free_in_bytes” : 28512256,
“used_in_bytes” : 4266455040,
“free_percent” : 1,
“used_percent” : 99
},
Output from kubectl top pod
eric-data-search-engine-data-3 888m 2622Mi

In both the examples, OpenSearch Stats API displaying high values in mem_used_bytes compared to kubectl top pods.

[radu.gheorghe]

Oh, sorry. I didn’t realize used_in_bytes also accounts for the buffered/cached memory. So anything that’s memory mapped will also show up in there. Which makes this stat… not terribly useful, unless your index size is supposed to fit in your OS cache (and this way you can check).

Also, this has nothing to do with containers. Here’s what I see on a VM:

    "mem" : {
      "total_in_bytes" : 66610937856,
      "free_in_bytes" : 823046144,
      "used_in_bytes" : 65787891712,
      "free_percent" : 1,
      "used_percent" : 99

Because I’m also monitoring it with Sematext Cloud (which collects most OS metrics from the OS itself, not the Stats API), I know for a fact that a big chunk of those 64GB are OS caches: