How to get actual memory (excluding cache/buffer memory) consumed by opensearch process

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 2.11.0

Describe the issue:
Mem used in Bytes from Stats API is showing higher than Kubectl top pods response - OpenSearch - OpenSearch

It is evident from the discussion above that “used_in_bytes” includes buffered/cached memory; nevertheless, whatever stat API allows us to obtain the real memory used by the opensearch process (apart from cache/buffer memory), which ought to be equal to the output of kubectl top pods?

Would you kindly let us know which stat API we should use to verify the actual memory?

Regards,
Chiranjeevi

Configuration:

Relevant Logs or Screenshots:

2

Hi,
Any updates on my request.

Regards,
Chiranjeevi

3

Hi,
Can someone help us on this request, pls.

4

Hi,
Any update on this concern?

5

Hi @chirumanem ,

Please share the output of your stat API request that you are using and kubectl top pods in your cluster.

How did you install your OpenSearch cluster? Do you have one container per pod?

6

Hi @Eugene7 ,

Sorry for the late response,
We are using OpenSearch 2.19.0 in our env and we have 2 data, 3 master and 1 ingest as default.

Kubectl Top Pods Output:

NAME                                                 CPU(cores)   MEMORY(bytes)
eric-data-search-engine-data-0                       128m         1567Mi
eric-data-search-engine-data-1                       149m         1562Mi
eric-data-search-engine-ingest-tls-746d9947d-s8r7n   71m          851Mi
eric-data-search-engine-master-0                     75m          856Mi
eric-data-search-engine-master-1                     72m          878Mi
eric-data-search-engine-master-2                     101m         908Mi

Output with " /_nodes/stats/os?pretty

eric-data-search-engine-data-0
      "os" : {
        "timestamp" : 1746014475642,
        "cpu" : {
          "percent" : 4,
          "load_average" : {
            "1m" : 1.68,
            "5m" : 2.59,
            "15m" : 2.71
          }
        },
        "mem" : {
          "total_in_bytes" : 2147483648,
          "free_in_bytes" : 105570304,
          "used_in_bytes" : 2041913344,
          "free_percent" : 5,
          "used_percent" : 95
        },
        "swap" : {
          "total_in_bytes" : 0,
          "free_in_bytes" : 0,
          "used_in_bytes" : 0
        },
        "cgroup" : {
          "cpuacct" : {
            "control_group" : "/",
            "usage_nanos" : 3402473240214
          },
          "cpu" : {
            "control_group" : "/",
            "cfs_period_micros" : 100000,
            "cfs_quota_micros" : 50000,
            "stat" : {
              "number_of_elapsed_periods" : 223093,
              "number_of_times_throttled" : 54170,
              "time_throttled_nanos" : 9621001926594
            }
          },
          "memory" : {
            "control_group" : "/",
            "limit_in_bytes" : "2147483648",
            "usage_in_bytes" : "2041913344"
          }
        }
      }

As you can see, the stat API displays “used_percent” of 95, while the kubectl top pods for data-0 display 1567 Mi. This issue was brought up by me a long time ago ( Mem used in Bytes from Stats API is showing higher than Kubectl top pods response - OpenSearch - OpenSearch .However, Mr. Radu Gheorghe highlighted at the time that the stat API contains buffer/cache memory, which cannot come up with kubectl top pods command (Mem used in Bytes from Stats API is showing higher than Kubectl top pods response).

We now want to know which stat API will provide us with the actual RAM, which ought to resemble the output from kubectl top.

Can you please help me out on this concern?

Regards,
Chiranjeevi

7

Hi @Eugene7 ,

Please let me know on how to check resident memory (actual memory consumed by opensearch i.e. without cache/buffer memory).

Regards,
Chiranjeevi

8

Hi @Eugene7 ,
Any updates on this concern?

Regards,
Chiranjeevi

9

Hi,
Can someone please respond on this concern.

Regards,
Chiranjeevi

10

Hi,
Any updates, please let me know

Regards