Is there any prefered method to monitor Opensearch regarding unsafe memory access faults?

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
2.13.0

Describe the issue:
In one of our clusters, we hit an error:

java.lang.InternalError: a fault occurred in an unsafe memory access operation

This forced us to have to restart the Opensearch service. Is there any tool or method that is prefered to monitor Opensearch to prevent something like this happening again? I see in posts that the root cause is due to some disk issue, but in our case, it looks like it stemmed from some corruption of files:

org.opensearch.bootstrap.StartupException: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 1970-01-01T00:00:00Z, (lock=NativeFSLock(path=/usr/share/opensearch/data/nodes/0/node.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2024-05-31T14:35:00.51631043Z))

We were not able to determine exactly why this happened so we are asking if anyone does any monitoring like this and can help us understand what to look for and why this can happen.

Thanks

2

@jpelletier_2023 please check for a similar issue, it may resolve your problem, thank you

[1] java.lang.InternalError: a fault occurred in an unsafe memory access operation

3

@reta Thanks for reply. We had reviewed this issue previously and were able to resolve the issue. We are now wondering if anyone is doing any monitoring of their Opensearch clusters to make sure this issue is not met and if so if they can share what they have implemented.