I have encountered the following problem. There is a cluster Opensearch, yesterday in consequence of lack of memory many nodes received a watermark, the cluster went into red status. 40% of shards were with unasigned shards.
Expanded the space, shards with red status cleaned up. But there was a problem with the .opendistro-anomaly-checkpoints system index.
it can’t bind the shard because it can’t find a master replica of the index.
‘allocate_explanation’ : ’cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster’
Probably because of this the cluster is still in red status. Although the indexes say 100%.
1729237118 07:38:38 opensearch red 13 13 true 11706 5892 0 0 0 2 0 - 100.0%
Judging by the documentation, you can try to remove this index from the list in the configuration and then delete it. But I have concerns that such a thing will only make the situation worse.
System indexes - OpenSearch Documentation.
Please, share your experience, maybe someone has already encountered something like this?
Translated with DeepL.com (free version)