Copy snapshot between buckets and restore

OpenSearch
1

Versions 3.3.1:

Describe the issue:

Until now I was using OpenSearch 2.10.0 and it was possible to:

  1. Create a snapshot in a GCS bucket named bucket1

  2. Copy the snapshot files with gsutil to another GCS bucket named bucket2

  3. Register bucket2 as a new repository and restore the snapshot from there

This was useful for DR scenarios.

After upgrading to OpenSearch 3.x this flow no longer works. Restoring from bucket2 fails with:

IndexShardRestoreFailedException[failed to restore snapshot [latest/WTNXuPysS3-ty09NvUohXw]]; 
nested: SnapshotMissingException[[bucket2:latest] is missing]

even though the snapshot exists in the bucket2 repository.

I noticed that the snapshot structure between 2.10.0 and 3.3.1 is different. The 3.3.1 snapshot includes an additional folder snapshot_shard_paths/ which contains metadata that seems to reference the original bucket.

My question: how can I restore from a copied snapshot in a different bucket when the original repository (original GCS bucket) is no longer available? Is there a supported way to use a “cloned” snapshot repository for DR in OpenSearch 3.x?

2

@yantsa Just to be clear, are you trying to restore a 2.10 snapshot on 3.3.1? Or did you make another snapshot with 3.3.1 and move to another 3.3.1 cluster?