Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.19; regression introduced between 2.16 and 2.19.
Describe the issue:
After upgrading from 2.16 to 2.19, our OpenSearch cluster has been periodically restarting (approximately every day) with the following error:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 2097152 bytes. Error detail: AllocateHeap
# An error report file with more information is saved as:
# /usr/share/opensearch/hs_err_pid10.log
JVM memory is within normal bounds so we suspect this failure is due to the max map count being reached. We’ve been setting vm.max_map_count to 262144 via sysctl and this has previously not caused issues. Have there been any changes between 2.16 and 2.19 that would have caused a regression here?
Configuration:
Cluster running 4 x {64 cpu, 512GB} nodes backed by GCP persistent disks. 102GB heap.
Relevant Logs or Screenshots:
Kubernetes memory is well below limit. We are unable to get the hs_error_log file that is dumped at crash time.
Let me know if there’s any other information that could help diagnose the issue.