Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch/Dashboard v 2.2.1 RHEL 7
Describe the issue:
Hi,
we have a 5-nodes cluster (with 3 master nodes) running daily snapshots policies for our indices.
We got a lot of warnings about snapshots only on one master node :
[WARN ][o.o.i.s.SMRunner ] [master-node] Cannot acquire lock for snapshot management job <name_policy>
[WARN ][o.o.i.s.SMRunner ] [master-node] Cannot acquire lock for snapshot management job <name_policy>
.....
[WARN ][o.o.r.b.BlobStoreRepository] [master-node] Your repository metadata blob for repository [name_repository] is larger than 5MB. Consider moving to a fresh repository for new snapshots or deleting unneeded snapshots from your repository to ensure stable repository behavior going forward.
then suddenly this master node left the cluster and we had problem ingesting data from logstash. We have lost some data then until we have restarted the node and disabled the snapshots policies.
How we can solve this issue? What can you suggest me?
Thanks in advance!
Regards,
Evelina