Restoring index from snapshot fails with snapshot_missing_exception exception

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

  • debian 11
  • 3x opensearch nodes (2.5.0 running in docker container)
  • snapshot repo via nfs mount into each opensearch docker container

Describe the issue:
I need to restore an index from our snapshot repository. We use daily-dated indices so restoring an old (not existing anymore) index should not require rename or delete an existing index
My restore curl command looks as follows

curl -X POST -k -H 'Content-Type: application/json' https://admin:redacted@localhost:9200/_snapshot/seppl-snap-repo/20230214_044701/_restore' -d'
 "indices": "redacted-2022-09-07",
 "partial": true,
 "ignore_unavailable": true,
 "include_global_state": false

the last two parameters I just added in the last try but it did not change anything. It still throws an error like this

"error":{"root_cause":[{"type":"snapshot_missing_exception","reason":"[seppl-snap-repo:20230214_044701/d64rfMCbTweXtBqRUrOZyA] is missing"}],"type":"snapshot_missing_exception","reason":"[seppl-snap-repo:20230214_044701/d64rfMCbTweXtBqRUrOZyA] is missing","caused_by":{"type":"no_such_file_exception","reason":"/backup/opensearch/indices/rucdkTYiSH2Y8n_8i_rbJQ/meta-Yru_noQB1TFRlOR8LLLM.dat"}},"status":404}

when checking the snapshot repo I can see that the mentioned meta-Yru_noQB1TFRlOR8LLLM.dat File is really missing in /backup/opensearch/indices/rucdkTYiSH2Y8n_8i_rbJQ/. We have a lot of files in that directory but all of them are named snap-*.dat

If I list the indices in snap 20230214_044701 I can see that the index I try to restore is contained in that snaphot

curl -X GET -k 'https://admin:redacted@localhost:9200/_snapshot/seppl-snap-repo/20230214_044701' 2>/dev/null|jq|grep redacted-2022-09-07


the snapshots are created every hour with the following bash script on the first opensearch node in cluster (always the same instance)


SNAP_NAME=$(date +'%Y%m%d_%H%M%S')
curl -k -X PUT "https://admin:redacted@localhost:9200/_snapshot/seppl-snap-repo/$SNAP_NAME?wait_for_completion=true" -H 'Content-Type: application/json' >/dev/null 2>&1
exit 0

Relevant Logs or Screenshots:
there are literally no additional loglines in the container’s log if a restore is called and fails

As not received any replies here I assume that our backup may not be restored at all. Can anyone see a reason what we did wrong in our snapshots? I mean having a regular backup which cannot be restored is nothing I would have expected from a snapshot function. Does anyone see a reason what we did wrong in our backups?