Snapshot Backward Compatibility for Minor Version Upgrades

DevOps
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): All versions up to 1.3.15 and 2.12.0

Describe the issue:
I have an OpenSearch cluster running version 1.3.11, preparing to upgrade to version 1.3.15. The below steps are what I did:

  1. Snapshot of all indices in the cluster with OpenSearch 1.3.11;

  2. Upgraded the whole cluster from OpenSearch 1.3.11 to 1.3.15;

  3. Snapshot with OpenSearch 1.3.15 to the same snapshot repository as step 1;

  4. Restored snapshot taken in step 3 to an OpenSearch 1.3.11 cluster – Failed;

  5. Restored snapshot taken in step 1 to an OpenSearch 1.3.11 cluster – Succeeded;

Although semantic versioning should ensure some sort of compatibility, I cannot restore to a new cluster in step 4.

[Question #1] Is this behavior expected? That OpenSearch does not guarantee snapshot backward compatibility between minor versions?

I cannot directly restore from newer version to older version like in step 4 but I can still manage to restore the last old-version snapshot like in step 5 in case of failed upgrade. This means the old version snapshot repository touched by a newer version cluster after upgrade like this is still restorable to an older version cluster. However, I have a concern:

[Question #2] Are older versions of OpenSearch guaranteed to be able to access a repository that has been modified by a newer version? Do I have to keep testing this behavior in future releases, and do we need another backup plan if it does not work?

I suppose, on a cluster with a single backup repository, Step 5 is the only way we can roll back to an older version in the event of failed upgrade (including unexpected breaking change). If what I’m concerned about in Question #2 is not guaranteed, I plan to use a separate snapshot repository for each minor version, which would be a lot to manage.

Although the strategy like in step 5 works, all changes after the upgrade (at step 2) such as creating, updating and deleting indices will surely be lost.

[Question #3] Is there any way to roll back to the older version that includes writes performed in the new version? Does not have to be an in-place rollback, restoring to a new cluster is fine as well.

If the behavior in step 5 always works,

[Question #4] is it worth documenting in the OpenSearch official documentation?

I have logged an issue in the GitHub repository of OpenSearch with somewhat similar questions.

Thank you very much.

2

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.