How to get actual memory (excluding cache/buffer memory) consumed by opensearch process

OpenSearch
6

Hi @Eugene7 ,

Sorry for the late response,
We are using OpenSearch 2.19.0 in our env and we have 2 data, 3 master and 1 ingest as default.

Kubectl Top Pods Output:

NAME                                                 CPU(cores)   MEMORY(bytes)
eric-data-search-engine-data-0                       128m         1567Mi
eric-data-search-engine-data-1                       149m         1562Mi
eric-data-search-engine-ingest-tls-746d9947d-s8r7n   71m          851Mi
eric-data-search-engine-master-0                     75m          856Mi
eric-data-search-engine-master-1                     72m          878Mi
eric-data-search-engine-master-2                     101m         908Mi

Output with " /_nodes/stats/os?pretty

eric-data-search-engine-data-0
      "os" : {
        "timestamp" : 1746014475642,
        "cpu" : {
          "percent" : 4,
          "load_average" : {
            "1m" : 1.68,
            "5m" : 2.59,
            "15m" : 2.71
          }
        },
        "mem" : {
          "total_in_bytes" : 2147483648,
          "free_in_bytes" : 105570304,
          "used_in_bytes" : 2041913344,
          "free_percent" : 5,
          "used_percent" : 95
        },
        "swap" : {
          "total_in_bytes" : 0,
          "free_in_bytes" : 0,
          "used_in_bytes" : 0
        },
        "cgroup" : {
          "cpuacct" : {
            "control_group" : "/",
            "usage_nanos" : 3402473240214
          },
          "cpu" : {
            "control_group" : "/",
            "cfs_period_micros" : 100000,
            "cfs_quota_micros" : 50000,
            "stat" : {
              "number_of_elapsed_periods" : 223093,
              "number_of_times_throttled" : 54170,
              "time_throttled_nanos" : 9621001926594
            }
          },
          "memory" : {
            "control_group" : "/",
            "limit_in_bytes" : "2147483648",
            "usage_in_bytes" : "2041913344"
          }
        }
      }

As you can see, the stat API displays “used_percent” of 95, while the kubectl top pods for data-0 display 1567 Mi. This issue was brought up by me a long time ago ( Mem used in Bytes from Stats API is showing higher than Kubectl top pods response - OpenSearch - OpenSearch .However, Mr. Radu Gheorghe highlighted at the time that the stat API contains buffer/cache memory, which cannot come up with kubectl top pods command (Mem used in Bytes from Stats API is showing higher than Kubectl top pods response).

We now want to know which stat API will provide us with the actual RAM, which ought to resemble the output from kubectl top.

Can you please help me out on this concern?

Regards,
Chiranjeevi