Cluster was running fine and randomly nodes losing connection

OpenSearch
1

Hi , i need urgent help if u cant listen me. I have my cluster 3.0 version. My cluster was running since 1 week without issue. Logstash was sending logs from openshift cluster to the opensearch cluster.

Always running good without error. yesterday randomly , my nodes start losing connection to the cluster continuously.

I see in the logs this

[2025-06-17T15:57:40,843][WARN ][i.n.c.AbstractChannelHandlerContext] [opensearch-master-02] An exception 'OpenSearchSecurityException[The provided TCP chann el is invalid.]; nested: DecoderException[javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more t han tag size (16)]; nested: SSLHandshakeException[Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)]; nested: B adPaddingException[Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)];' [enable DEBUG level for full stacktrace ] was thrown by a user handler's exceptionCaught() method while handling the following exception: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more th an tag size (16)

i don’t know how to resolve, i don’t change anything in the configuration. I don’t know if the logs is related about OOM or something else.

My cluster is composed by 9 datanodes (64 GB ram and 12 TB storage )and 3 master (16 GB ram)

My cluster was around 25% allocation.

Help me if u can. This is urgent

2

@abarocco Do you know if the certificates were auto-rotated perhaps?

Also you should check the Opensearch garbage collection logs for any GC pauses or memory pressure.

Also verify node memory pressure on individual nodes with tools like “dmesg” or “free”.

Lastly how much heap is assigned to the nodes?

3

hello @Anthony thanks for the reply.
I don’t have auto-rotated certificates.

I check the GC logs and this is what i have

[2025-06-18T12:32:10.759+0000][3530396][gc,start    ] GC(522) Pause Young (Normal) (G1 Evacuation Pause)
[2025-06-18T12:32:10.759+0000][3530396][gc,task     ] GC(522) Using 4 workers of 4 for evacuation
[2025-06-18T12:32:10.759+0000][3530396][gc,age      ] GC(522) Desired survivor size 322961408 bytes, new threshold 15 (max threshold 15)
[2025-06-18T12:32:10.770+0000][3530396][gc,phases   ] GC(522)   Pre Evacuate Collection Set: 0.4ms
[2025-06-18T12:32:10.770+0000][3530396][gc,phases   ] GC(522)   Merge Heap Roots: 0.3ms
[2025-06-18T12:32:10.770+0000][3530396][gc,phases   ] GC(522)   Evacuate Collection Set: 7.7ms
[2025-06-18T12:32:10.770+0000][3530396][gc,phases   ] GC(522)   Post Evacuate Collection Set: 1.8ms
[2025-06-18T12:32:10.770+0000][3530396][gc,phases   ] GC(522)   Other: 0.1ms
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) Age table with threshold 15 (max threshold 15)
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age   1:    1144104 bytes,    1144104 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age   2:    1095040 bytes,    2239144 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age   3:    1057280 bytes,    3296424 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age   4:    1049120 bytes,    4345544 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age   5:    1233008 bytes,    5578552 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age   6:      40880 bytes,    5619432 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age   7:    4154280 bytes,    9773712 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age   8:      20040 bytes,    9793752 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age   9:     142992 bytes,    9936744 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age  10:     297472 bytes,   10234216 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age  11:     657784 bytes,   10892000 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age  12:     654416 bytes,   11546416 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age  13:     737984 bytes,   12284400 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age  14:     638928 bytes,   12923328 total
[2025-06-18T12:32:10.770+0000][3530396][gc,age      ] GC(522) - age  15:     735808 bytes,   13659136 total
[2025-06-18T12:32:10.770+0000][3530396][gc,heap     ] GC(522) Eden regions: 1224->0(1224)
[2025-06-18T12:32:10.770+0000][3530396][gc,heap     ] GC(522) Survivor regions: 4->4(154)
[2025-06-18T12:32:10.770+0000][3530396][gc,heap     ] GC(522) Old regions: 190->191
[2025-06-18T12:32:10.770+0000][3530396][gc,heap     ] GC(522) Humongous regions: 2->2
[2025-06-18T12:32:10.770+0000][3530396][gc,metaspace] GC(522) Metaspace: 170682K(174336K)->170682K(174336K) NonClass: 149891K(151744K)->149891K(151744K) Class: 20791K(22592K)->20791K(22592K)
[2025-06-18T12:32:10.770+0000][3530396][gc          ] GC(522) Pause Young (Normal) (G1 Evacuation Pause) 5677M->784M(8192M) 10.667ms
[2025-06-18T12:32:10.770+0000][3530396][gc,cpu      ] GC(522) User=0.04s Sys=0.00s Real=0.01s
[2025-06-18T12:32:10.770+0000][3530396][safepoint   ] Safepoint "G1CollectForAllocation", Time since last: 5547621622 ns, Reaching safepoint: 11074 ns, Cleanup: 6335 ns, At safepoint: 10751440 ns, Total: 10768849 ns

On the data node i setup 32GB HEAP SIZE (total VM 64gb) And on the masters node 8GB (Total 16GB)

4

This appears to be in order, If there were indeed no changes in configuration, this can likely be caused by clock drift, I would recommend to ensure the nodes are synchronized.

Lastly of course this could be a result of packets being dropped over the network, and therefore nodes being unable to properly decrypt the packets.

5

Ok thanks you

6

@Anthony So u suggest to check if there are network problem or something else?
There si a specific group in my company that work on the network. What details can I provide to make the best possible network check?

7

i also check if the nodes are not synchronized each other.
This is the output of 2 nodes chronyd tracking

VM 1

Reference ID    : 0AB0011A (VMGCLALTE1088.syssede.systest.spi.com)
Stratum         : 3
Ref time (UTC)  : Wed Jun 18 13:09:39 2025
System time     : 0.000001429 seconds slow of NTP time
Last offset     : +0.000037964 seconds
RMS offset      : 0.000025977 seconds
Frequency       : 1.282 ppm fast
Residual freq   : +0.000 ppm
Skew            : 0.007 ppm
Root delay      : 0.008408082 seconds
Root dispersion : 0.001250133 seconds
Update interval : 1040.4 seconds
Leap status     : Normal

VM 2

Reference ID    : 0AB0011A (VMGCLALTE1088.syssede.systest.spi.com)
Stratum         : 3
Ref time (UTC)  : Wed Jun 18 12:59:12 2025
System time     : 0.000012626 seconds slow of NTP time
Last offset     : -0.000029539 seconds
RMS offset      : 0.000023536 seconds
Frequency       : 87.303 ppm slow
Residual freq   : -0.000 ppm
Skew            : 0.007 ppm
Root delay      : 0.007700196 seconds
Root dispersion : 0.001777516 seconds
Update interval : 1024.3 seconds
Leap status     : Normal
8

Based on the above the nodes appear to be synchronised. I would recommend to reach out to networking team asking to confirm if any changes have been done to the following:

Any SSL/TLS interception or DPI
Firewall/NAT timeouts or TCP session resets
MTU (Maximum Transmission Unit) changes (unlikely)

If there is anything running between the nodes, it would be good to bypass it and see if the connections gets fixed, this will enable you to diagnose the issue better.

also if they can confirm if they are seeing packet loss, retransmits, or routing issues between nodes.

9

Hi @Anthony the network team said they don’t see any packet loss issues or anything like that.
The weird thing is that if I don’t do anything on the cluster it works fine. As soon as I enter and launch a query from the dev tool like a very simple cluster_health, it takes 20k ms to respond and after a while I start losing shards and the nodes leave the cluster. From the opensearch logs when this happens, it always starts logging this message

[2025-06-23T10:23:10,703][WARN ][i.n.c.AbstractChannelHandlerContext] [opensearch-master-01] An exception 'OpenSearchSecurityException[The provided TCP cha
nnel is invalid.]; nested: DecoderException[javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be mo
re than tag size (16)]; nested: SSLHandshakeException[Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)]; nes
ted: BadPaddingException[Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)];' [enable DEBUG level for full st
acktrace] was thrown by a user handler's exceptionCaught() method while handling the following exception:
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more
than tag size (16)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:500) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:383) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326) ~[?:?]
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
        at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:134) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.JdkSslEngine.unwrap(JdkSslEngine.java:92) ~[?:?]
        at io.netty.handler.ssl.JdkAlpnSslEngine.unwrap(JdkAlpnSslEngine.java:163) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:309) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1485) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469) ~[?:?]

Any other suggests?

10

@pablo Can u help me too?

11

@abarocco It seems there is already an open issue for this. Some users have reported disabling hostname verification as a temporary fix. Have you tried this?

12

Yes i have the hostname verification false in all my opensearch.yml

13

@abarocco can you please provide your opensearch.yml file also the uptime of off the nodes, using GET /_nodes/stats/jvm. It is very strange that this broke after working for a week.

14

@Anthony Sure, this is the yaml of the master node :

cluster.name: opensearch-moncalieri
node.name: opensearch-master-01
node.roles: [ cluster_manager ]
network.host: 0.0.0.0

path.data: "/mnt/opensearch"
path.logs: "/var/log/opensearch"

discovery.seed_hosts: ["10.174.110.159", "10.174.110.149", "10.174.110.124"]
#cluster.initial_cluster_manager_nodes: ["10.174.110.159", "10.174.110.149", "10.174.110.124"]

plugins.security.ssl.transport.enabled: true
plugins.security.ssl.transport.pemcert_filepath: /etc/opensearch/certs/vmgclalte1757.syssede.systest.sanpaoloimi.com.cer
plugins.security.ssl.transport.pemkey_filepath: /etc/opensearch/certs/vmgclalte1757.key
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/opensearch/certs/chain.pem
plugins.security.ssl.transport.enforce_hostname_verification: false

plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: /etc/opensearch/certs/vmgclalte1757.syssede.systest.sanpaoloimi.com.cer
plugins.security.ssl.http.pemkey_filepath: /etc/opensearch/certs/vmgclalte1757.key
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/opensearch/certs/chain.pem

plugins.security.authcz.admin_dn:
 - "CN=admin,OU=xx,O=xx,L=xx,ST=xx,C=xx"
plugins.security.nodes_dn:
  - 'CN=*xx,OU=xx.,O=xx,L=xx,ST=xx,C=xx'

plugins.security.allow_default_init_securityindex: true
plugins.security.restapi.roles_enabled: ["all_access"]
bootstrap.system_call_filter: false
bootstrap.memory_lock: true
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_DHE_RSA_WITH_AES_256_CBC_SHA"
  - "TLS_DHE_DSS_WITH_AES_128_CBC_SHA256"
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.1"
  - "TLSv1.2"

For This settings I tried to put them to solve the problem, reading about some open cases but they didn’t solve anything

plugins.security.ssl.http.enabled_ciphers:

  • “TLS_DHE_RSA_WITH_AES_256_CBC_SHA”
  • “TLS_DHE_DSS_WITH_AES_128_CBC_SHA256”
    plugins.security.ssl.http.enabled_protocols:
  • “TLSv1.1”
  • “TLSv1.2”

3 Uptime of 1 master and 2 data nodes. If u want all the uptime of the 12 vm tell me.

[root@vmgclalte1757 /etc/opensearch]# uptime
 09:30:50 up 39 days, 23:19,  1 user,  load average: 0.06, 0.12, 0.14

[root@vmgclalte1745 ~]# uptime
 09:31:15 up 39 days, 23:41,  1 user,  load average: 3.59, 3.38, 3.21

[root@vmgclalte1747 ~]# uptime
 09:31:41 up 39 days, 23:38,  1 user,  load average: 2.74, 2.78, 3.17

This is the stats jvm

{
  "_nodes": {
    "total": 12,
    "successful": 12,
    "failed": 0
  },
  "cluster_name": "opensearch-moncalieri",
  "nodes": {
    "4j7tOj24QkW4RYTXvDuZWA": {
      "timestamp": 1750750403031,
      "name": "opensearch-data-07",
      "transport_address": "10.174.110.133:9300",
      "host": "10.174.110.133",
      "ip": "10.174.110.133:9300",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403031,
        "uptime_in_millis": 320476670,
        "mem": {
          "heap_used_in_bytes": 13293184544,
          "heap_used_percent": 38,
          "heap_committed_in_bytes": 34359738368,
          "heap_max_in_bytes": 34359738368,
          "non_heap_used_in_bytes": 270427952,
          "non_heap_committed_in_bytes": 315817984,
          "pools": {
            "young": {
              "used_in_bytes": 2399141888,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 20401094656,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 9677694496,
              "max_in_bytes": 34359738368,
              "peak_used_in_bytes": 10850497536,
              "peak_max_in_bytes": 34359738368,
              "last_gc_stats": {
                "used_in_bytes": 9677694496,
                "max_in_bytes": 34359738368,
                "usage_percent": 28
              }
            },
            "survivor": {
              "used_in_bytes": 1216348160,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 1577058304,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 1216348160,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 120,
          "peak_count": 186
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 9014,
              "collection_time_in_millis": 449314
            },
            "G1 Concurrent GC": {
              "collection_count": 5986,
              "collection_time_in_millis": 18372
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 30390,
            "used_in_bytes": 2950168583841,
            "total_capacity_in_bytes": 2950168583841
          },
          "direct": {
            "count": 113,
            "used_in_bytes": 19161943,
            "total_capacity_in_bytes": 19161942
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33232,
          "total_loaded_count": 35221,
          "total_unloaded_count": 1989
        }
      }
    },
    "FAE10zJUTd6IzBteDxd4Fg": {
      "timestamp": 1750750403031,
      "name": "opensearch-data-02",
      "transport_address": "10.174.110.248:9300",
      "host": "10.174.110.248",
      "ip": "10.174.110.248:9300",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403031,
        "uptime_in_millis": 321008126,
        "mem": {
          "heap_used_in_bytes": 10771266384,
          "heap_used_percent": 31,
          "heap_committed_in_bytes": 34359738368,
          "heap_max_in_bytes": 34359738368,
          "non_heap_used_in_bytes": 269208592,
          "non_heap_committed_in_bytes": 318570496,
          "pools": {
            "young": {
              "used_in_bytes": 134217728,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 20468203520,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 9353591632,
              "max_in_bytes": 34359738368,
              "peak_used_in_bytes": 10810904912,
              "peak_max_in_bytes": 34359738368,
              "last_gc_stats": {
                "used_in_bytes": 9293264736,
                "max_in_bytes": 34359738368,
                "usage_percent": 27
              }
            },
            "survivor": {
              "used_in_bytes": 1283457024,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 1610612736,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 1283457024,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 125,
          "peak_count": 188
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 8216,
              "collection_time_in_millis": 350862
            },
            "G1 Concurrent GC": {
              "collection_count": 5446,
              "collection_time_in_millis": 13110
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 30550,
            "used_in_bytes": 2789249497254,
            "total_capacity_in_bytes": 2789249497254
          },
          "direct": {
            "count": 116,
            "used_in_bytes": 19206458,
            "total_capacity_in_bytes": 19206457
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33260,
          "total_loaded_count": 35187,
          "total_unloaded_count": 1927
        }
      }
    },
    "bTaA_568RwGsSFkiK3aa7w": {
      "timestamp": 1750750403030,
      "name": "opensearch-data-06",
      "transport_address": "10.174.110.245:9300",
      "host": "10.174.110.245",
      "ip": "10.174.110.245:9300",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403030,
        "uptime_in_millis": 320558108,
        "mem": {
          "heap_used_in_bytes": 24066721976,
          "heap_used_percent": 70,
          "heap_committed_in_bytes": 34359738368,
          "heap_max_in_bytes": 34359738368,
          "non_heap_used_in_bytes": 288389208,
          "non_heap_committed_in_bytes": 331350016,
          "pools": {
            "young": {
              "used_in_bytes": 13589544960,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 20468203520,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 9445378232,
              "max_in_bytes": 34359738368,
              "peak_used_in_bytes": 10048483688,
              "peak_max_in_bytes": 34359738368,
              "last_gc_stats": {
                "used_in_bytes": 9445378232,
                "max_in_bytes": 34359738368,
                "usage_percent": 27
              }
            },
            "survivor": {
              "used_in_bytes": 1031798784,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 1593835520,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 1031798784,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 154,
          "peak_count": 219
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 9386,
              "collection_time_in_millis": 384567
            },
            "G1 Concurrent GC": {
              "collection_count": 6244,
              "collection_time_in_millis": 16803
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 32528,
            "used_in_bytes": 2952833971637,
            "total_capacity_in_bytes": 2952833971637
          },
          "direct": {
            "count": 125,
            "used_in_bytes": 19449268,
            "total_capacity_in_bytes": 19449267
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33885,
          "total_loaded_count": 35761,
          "total_unloaded_count": 1876
        }
      }
    },
    "a9HyeEZSQiCHj82QGQcWTQ": {
      "timestamp": 1750750429437,
      "name": "opensearch-data-04",
      "transport_address": "10.174.110.30:9300",
      "host": "10.174.110.30",
      "ip": "10.174.110.30:9300",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750429437,
        "uptime_in_millis": 320818226,
        "mem": {
          "heap_used_in_bytes": 19075694592,
          "heap_used_percent": 55,
          "heap_committed_in_bytes": 34359738368,
          "heap_max_in_bytes": 34359738368,
          "non_heap_used_in_bytes": 270166952,
          "non_heap_committed_in_bytes": 316145664,
          "pools": {
            "young": {
              "used_in_bytes": 8304721920,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 20468203520,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 9781116928,
              "max_in_bytes": 34359738368,
              "peak_used_in_bytes": 10284433408,
              "peak_max_in_bytes": 34359738368,
              "last_gc_stats": {
                "used_in_bytes": 9781116928,
                "max_in_bytes": 34359738368,
                "usage_percent": 28
              }
            },
            "survivor": {
              "used_in_bytes": 989855744,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 1610612736,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 989855744,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 133,
          "peak_count": 202
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 9397,
              "collection_time_in_millis": 421950
            },
            "G1 Concurrent GC": {
              "collection_count": 6250,
              "collection_time_in_millis": 17279
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 31231,
            "used_in_bytes": 2985686884486,
            "total_capacity_in_bytes": 2985686884486
          },
          "direct": {
            "count": 109,
            "used_in_bytes": 19050820,
            "total_capacity_in_bytes": 19050819
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33407,
          "total_loaded_count": 35360,
          "total_unloaded_count": 1953
        }
      }
    },
    "BoDvbe8eSa-ygY8gDlqBtA": {
      "timestamp": 1750750403031,
      "name": "opensearch-data-01",
      "transport_address": "10.174.110.141:9300",
      "host": "10.174.110.141",
      "ip": "10.174.110.141:9300",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403031,
        "uptime_in_millis": 321091030,
        "mem": {
          "heap_used_in_bytes": 23898449344,
          "heap_used_percent": 69,
          "heap_committed_in_bytes": 34359738368,
          "heap_max_in_bytes": 34359738368,
          "non_heap_used_in_bytes": 271061120,
          "non_heap_committed_in_bytes": 317980672,
          "pools": {
            "young": {
              "used_in_bytes": 12968787968,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 20367540224,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 9805587904,
              "max_in_bytes": 34359738368,
              "peak_used_in_bytes": 10636754944,
              "peak_max_in_bytes": 34359738368,
              "last_gc_stats": {
                "used_in_bytes": 9805587904,
                "max_in_bytes": 34359738368,
                "usage_percent": 28
              }
            },
            "survivor": {
              "used_in_bytes": 1124073472,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 1602224128,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 1124073472,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 144,
          "peak_count": 203
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 8450,
              "collection_time_in_millis": 388255
            },
            "G1 Concurrent GC": {
              "collection_count": 5598,
              "collection_time_in_millis": 17198
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 31446,
            "used_in_bytes": 3095616201060,
            "total_capacity_in_bytes": 3095616201060
          },
          "direct": {
            "count": 122,
            "used_in_bytes": 19273013,
            "total_capacity_in_bytes": 19273012
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33347,
          "total_loaded_count": 35159,
          "total_unloaded_count": 1812
        }
      }
    },
    "o3zlfuhQTsKAPa16MuR91A": {
      "timestamp": 1750750403030,
      "name": "opensearch-data-05",
      "transport_address": "10.174.110.156:9300",
      "host": "10.174.110.156",
      "ip": "10.174.110.156:9300",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403031,
        "uptime_in_millis": 320688674,
        "mem": {
          "heap_used_in_bytes": 17831620464,
          "heap_used_percent": 51,
          "heap_committed_in_bytes": 34359738368,
          "heap_max_in_bytes": 34359738368,
          "non_heap_used_in_bytes": 285964880,
          "non_heap_committed_in_bytes": 336265216,
          "pools": {
            "young": {
              "used_in_bytes": 7214202880,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 20468203520,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 9350737776,
              "max_in_bytes": 34359738368,
              "peak_used_in_bytes": 10699905216,
              "peak_max_in_bytes": 34359738368,
              "last_gc_stats": {
                "used_in_bytes": 9350737776,
                "max_in_bytes": 34359738368,
                "usage_percent": 27
              }
            },
            "survivor": {
              "used_in_bytes": 1266679808,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 1560281088,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 1266679808,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 159,
          "peak_count": 223
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 8675,
              "collection_time_in_millis": 372114
            },
            "G1 Concurrent GC": {
              "collection_count": 5752,
              "collection_time_in_millis": 18137
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 31914,
            "used_in_bytes": 2741626743637,
            "total_capacity_in_bytes": 2741626743637
          },
          "direct": {
            "count": 137,
            "used_in_bytes": 19663134,
            "total_capacity_in_bytes": 19663133
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 34082,
          "total_loaded_count": 36049,
          "total_unloaded_count": 1967
        }
      }
    },
    "TsBSTCBNSTaAgeesSoe2Kw": {
      "timestamp": 1750750403031,
      "name": "opensearch-data-03",
      "transport_address": "10.174.110.28:9300",
      "host": "10.174.110.28",
      "ip": "10.174.110.28:9300",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403031,
        "uptime_in_millis": 320889646,
        "mem": {
          "heap_used_in_bytes": 23614380144,
          "heap_used_percent": 68,
          "heap_committed_in_bytes": 34359738368,
          "heap_max_in_bytes": 34359738368,
          "non_heap_used_in_bytes": 269782968,
          "non_heap_committed_in_bytes": 316866560,
          "pools": {
            "young": {
              "used_in_bytes": 12280922112,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 20484980736,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 10175830128,
              "max_in_bytes": 34359738368,
              "peak_used_in_bytes": 10767798624,
              "peak_max_in_bytes": 34359738368,
              "last_gc_stats": {
                "used_in_bytes": 10175830128,
                "max_in_bytes": 34359738368,
                "usage_percent": 29
              }
            },
            "survivor": {
              "used_in_bytes": 1157627904,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 1535115264,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 1157627904,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 142,
          "peak_count": 209
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 9945,
              "collection_time_in_millis": 481966
            },
            "G1 Concurrent GC": {
              "collection_count": 6598,
              "collection_time_in_millis": 18471
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 29013,
            "used_in_bytes": 3205226009648,
            "total_capacity_in_bytes": 3205226009648
          },
          "direct": {
            "count": 115,
            "used_in_bytes": 19082728,
            "total_capacity_in_bytes": 19082727
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33329,
          "total_loaded_count": 35281,
          "total_unloaded_count": 1952
        }
      }
    },
    "D792IMHHRBW4wpweAmTSYA": {
      "timestamp": 1750750403031,
      "name": "opensearch-master-03",
      "transport_address": "10.174.110.124:9300",
      "host": "10.174.110.124",
      "ip": "10.174.110.124:9300",
      "roles": [
        "cluster_manager"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403031,
        "uptime_in_millis": 321289286,
        "mem": {
          "heap_used_in_bytes": 5482755072,
          "heap_used_percent": 63,
          "heap_committed_in_bytes": 8589934592,
          "heap_max_in_bytes": 8589934592,
          "non_heap_used_in_bytes": 257318216,
          "non_heap_committed_in_bytes": 299040768,
          "pools": {
            "young": {
              "used_in_bytes": 4676648960,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 5142216704,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 779816760,
              "max_in_bytes": 8589934592,
              "peak_used_in_bytes": 918552576,
              "peak_max_in_bytes": 8589934592,
              "last_gc_stats": {
                "used_in_bytes": 723598392,
                "max_in_bytes": 8589934592,
                "usage_percent": 8
              }
            },
            "survivor": {
              "used_in_bytes": 26289352,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 314572800,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 26289352,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 62,
          "peak_count": 64
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 1472,
              "collection_time_in_millis": 44416
            },
            "G1 Concurrent GC": {
              "collection_count": 112,
              "collection_time_in_millis": 737
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          },
          "direct": {
            "count": 26,
            "used_in_bytes": 4450112,
            "total_capacity_in_bytes": 4450111
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33237,
          "total_loaded_count": 34973,
          "total_unloaded_count": 1736
        }
      }
    },
    "Pr2p0DLiSdypaADyoh3wyg": {
      "timestamp": 1750750403030,
      "name": "opensearch-master-01",
      "transport_address": "10.174.110.159:9300",
      "host": "10.174.110.159",
      "ip": "10.174.110.159:9300",
      "roles": [
        "cluster_manager"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403031,
        "uptime_in_millis": 321951075,
        "mem": {
          "heap_used_in_bytes": 4445822976,
          "heap_used_percent": 51,
          "heap_committed_in_bytes": 8589934592,
          "heap_max_in_bytes": 8589934592,
          "non_heap_used_in_bytes": 248024456,
          "non_heap_committed_in_bytes": 292683776,
          "pools": {
            "young": {
              "used_in_bytes": 3938451456,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 5146411008,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 500054360,
              "max_in_bytes": 8589934592,
              "peak_used_in_bytes": 1327497216,
              "peak_max_in_bytes": 8589934592,
              "last_gc_stats": {
                "used_in_bytes": 500054360,
                "max_in_bytes": 8589934592,
                "usage_percent": 5
              }
            },
            "survivor": {
              "used_in_bytes": 7317160,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 559939584,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 7317160,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 61,
          "peak_count": 64
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 2514,
              "collection_time_in_millis": 37875
            },
            "G1 Concurrent GC": {
              "collection_count": 298,
              "collection_time_in_millis": 2012
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          },
          "direct": {
            "count": 28,
            "used_in_bytes": 4450391,
            "total_capacity_in_bytes": 4450390
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33435,
          "total_loaded_count": 35223,
          "total_unloaded_count": 1788
        }
      }
    },
    "oLSayB1jRSqCXt8o97f0eA": {
      "timestamp": 1750750403031,
      "name": "opensearch-data-09",
      "transport_address": "10.174.110.167:9300",
      "host": "10.174.110.167",
      "ip": "10.174.110.167:9300",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403031,
        "uptime_in_millis": 320253565,
        "mem": {
          "heap_used_in_bytes": 23673439920,
          "heap_used_percent": 68,
          "heap_committed_in_bytes": 34359738368,
          "heap_max_in_bytes": 34359738368,
          "non_heap_used_in_bytes": 272056872,
          "non_heap_committed_in_bytes": 314376192,
          "pools": {
            "young": {
              "used_in_bytes": 12717129728,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 20417871872,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 9958065840,
              "max_in_bytes": 34359738368,
              "peak_used_in_bytes": 11151364168,
              "peak_max_in_bytes": 34359738368,
              "last_gc_stats": {
                "used_in_bytes": 9958065840,
                "max_in_bytes": 34359738368,
                "usage_percent": 28
              }
            },
            "survivor": {
              "used_in_bytes": 998244352,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 1577058304,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 998244352,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 154,
          "peak_count": 211
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 10892,
              "collection_time_in_millis": 515623
            },
            "G1 Concurrent GC": {
              "collection_count": 7242,
              "collection_time_in_millis": 19518
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 30973,
            "used_in_bytes": 3525595608867,
            "total_capacity_in_bytes": 3525595608867
          },
          "direct": {
            "count": 128,
            "used_in_bytes": 19131207,
            "total_capacity_in_bytes": 19131206
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33603,
          "total_loaded_count": 35636,
          "total_unloaded_count": 2033
        }
      }
    },
    "735qI6E1QfyICwx6Gq4HDw": {
      "timestamp": 1750750403032,
      "name": "opensearch-data-08",
      "transport_address": "10.174.110.44:9300",
      "host": "10.174.110.44",
      "ip": "10.174.110.44:9300",
      "roles": [
        "data",
        "ingest"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403032,
        "uptime_in_millis": 320321211,
        "mem": {
          "heap_used_in_bytes": 21251429280,
          "heap_used_percent": 61,
          "heap_committed_in_bytes": 34359738368,
          "heap_max_in_bytes": 34359738368,
          "non_heap_used_in_bytes": 272286864,
          "non_heap_committed_in_bytes": 316276736,
          "pools": {
            "young": {
              "used_in_bytes": 9965666304,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 20468203520,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 10228798368,
              "max_in_bytes": 34359738368,
              "peak_used_in_bytes": 11307843584,
              "peak_max_in_bytes": 34359738368,
              "last_gc_stats": {
                "used_in_bytes": 10228798368,
                "max_in_bytes": 34359738368,
                "usage_percent": 29
              }
            },
            "survivor": {
              "used_in_bytes": 1056964608,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 1577058304,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 1056964608,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 152,
          "peak_count": 222
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 11598,
              "collection_time_in_millis": 612580
            },
            "G1 Concurrent GC": {
              "collection_count": 7706,
              "collection_time_in_millis": 23774
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 30918,
            "used_in_bytes": 3822369114714,
            "total_capacity_in_bytes": 3822369114714
          },
          "direct": {
            "count": 124,
            "used_in_bytes": 19279984,
            "total_capacity_in_bytes": 19279983
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 33406,
          "total_loaded_count": 35349,
          "total_unloaded_count": 1943
        }
      }
    },
    "BW_riyEBSPaaWxvUL7hhZw": {
      "timestamp": 1750750403032,
      "name": "opensearch-master-02",
      "transport_address": "10.174.110.149:9300",
      "host": "10.174.110.149",
      "ip": "10.174.110.149:9300",
      "roles": [
        "cluster_manager"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "jvm": {
        "timestamp": 1750750403032,
        "uptime_in_millis": 321494816,
        "mem": {
          "heap_used_in_bytes": 5836116744,
          "heap_used_percent": 67,
          "heap_committed_in_bytes": 8589934592,
          "heap_max_in_bytes": 8589934592,
          "non_heap_used_in_bytes": 240327632,
          "non_heap_committed_in_bytes": 293797888,
          "pools": {
            "young": {
              "used_in_bytes": 5117050880,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 5146411008,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 0,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            },
            "old": {
              "used_in_bytes": 712335560,
              "max_in_bytes": 8589934592,
              "peak_used_in_bytes": 1193279488,
              "peak_max_in_bytes": 8589934592,
              "last_gc_stats": {
                "used_in_bytes": 693658848,
                "max_in_bytes": 8589934592,
                "usage_percent": 8
              }
            },
            "survivor": {
              "used_in_bytes": 6730304,
              "max_in_bytes": 0,
              "peak_used_in_bytes": 383778816,
              "peak_max_in_bytes": 0,
              "last_gc_stats": {
                "used_in_bytes": 6730304,
                "max_in_bytes": 0,
                "usage_percent": -1
              }
            }
          }
        },
        "threads": {
          "count": 60,
          "peak_count": 62
        },
        "gc": {
          "collectors": {
            "young": {
              "collection_count": 1327,
              "collection_time_in_millis": 22495
            },
            "G1 Concurrent GC": {
              "collection_count": 32,
              "collection_time_in_millis": 384
            },
            "old": {
              "collection_count": 0,
              "collection_time_in_millis": 0
            }
          }
        },
        "buffer_pools": {
          "mapped": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          },
          "direct": {
            "count": 25,
            "used_in_bytes": 4433728,
            "total_capacity_in_bytes": 4433727
          },
          "mapped - 'non-volatile memory'": {
            "count": 0,
            "used_in_bytes": 0,
            "total_capacity_in_bytes": 0
          }
        },
        "classes": {
          "current_loaded_count": 32700,
          "total_loaded_count": 34440,
          "total_unloaded_count": 1740
        }
      }
    }
  }
}

For this query , it took 25456ms, too much for a simple query like this one.

15

After this query and a _cluster/health. This is the situation

{
  "cluster_name": "opensearch-moncalieri",
  "status": "red",
  "timed_out": false,
  "number_of_nodes": 12,
  "number_of_data_nodes": 9,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 3623,
  "active_shards": 4000,
  "relocating_shards": 0,
  "initializing_shards": 4,
  "unassigned_shards": 7066,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 15,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 59915,
  "active_shards_percent_as_number": 36.133694670280036
}
16

@Anthony U think this problem can be related by an JDK issue? I’ve tried to downgrade JDK version to the 17 but can’t start opensearch because OpenSearch code has been compiled by JDK 21

17

How did you create the certificates for this cluster?

18 
[req]
distinguished_name = req_distinguished_name
req_extensions     = req_ext
x509_extensions    = v3_req
prompt             = no

[req_distinguished_name]
C  = XX
ST = XX
L  = XX
O  = XX
OU = XX
CN = hostname

[req_ext]
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = @alt_names

[v3_req]
nsCertType = client, server
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names

[alt_names]
DNS.0 = hostname

I’ve used this files to create the CSR , then i send the CSR to the company and they give me the Certificate signed by the company CA

19

Can you share for the 12 nodes please.

20 
MASTER 1
 13:12:29 up 40 days,  3:01,  1 user,  load average: 0.64, 1.93, 1.92

MASTER 2 
 13:12:58 up 40 days,  3:00,  1 user,  load average: 1.87, 2.04, 1.74

MASTER 3
 13:13:25 up 40 days,  3:03,  1 user,  load average: 0.95, 1.36, 1.26

DATA NODE 1
 13:13:50 up 40 days,  3:23,  1 user,  load average: 0.29, 1.21, 1.40

DATA NODE 2
 13:14:15 up 40 days,  3:18,  1 user,  load average: 2.03, 1.67, 1.29

DATA NODE 3
 13:14:36 up 40 days,  3:21,  1 user,  load average: 1.62, 0.99, 1.17

DATA NODE 4
 13:14:55 up 40 days,  3:21,  1 user,  load average: 1.89, 1.09, 1.21

DATA NODE 5
 13:15:15 up 40 days,  3:17,  1 user,  load average: 2.29, 1.14, 0.78

DATA NODE 6
 13:15:39 up 40 days,  3:15,  1 user,  load average: 1.00, 1.59, 1.59

DATA NODE 7
 13:16:03 up 40 days,  3:13,  1 user,  load average: 1.51, 1.69, 1.52

DATA NODE 8
 13:16:31 up 40 days,  3:25,  1 user,  load average: 0.94, 0.88, 1.06

DATA NODE 9
 13:16:55 up 40 days,  3:19,  1 user,  load average: 0.84, 1.20, 1.38