The docs say “For example, if cluster health goes red, you might restore the red indices from a snapshot.”
I am running an AWS hosted cluster, which says it takes hourly automated snapshots. I attempted to restore a snapshot that was too big on my dev cluster, and it ran out of storage space. I tried to resize the storage space, but it got stuck in the processing state.
Amazon OpenSearch Service domain is stuck in "Processing" | AWS re:Post says to contact AWS Support if this happens. They responded that my account was not set up for support properly (support associated with the production account, not the dev account, doesn’t matter that they have the same billing entity) and they couldn’t help.
I thought I could start a new cluster in the same account and restore an automated snapshot from the failed cluster (before it failed). The automated snapshots go to a hidden S3 bucket that’s not accessible or visible to me. I tried GET _snapshot/_all from the new cluster, but it did not show any of the automated snapshots from the old cluster.
My disaster recovery was a disaster: I couldn’t find anything about how to actually use the automated snapshots when I needed them, and AWS support did not help. What am I missing? Do I need to set up my own periodic manual snapshots to be able to recover a failed cluster?