OpenSearch 3.0 Upgrade Issue: System Indices with 7.x Created Version But 2.x Upgraded Version

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
OpenSearch 2.19.2
OpenSearch 3.0.0

Describe the issue:
During an upgrade from OpenSearch 2.19.2 to 3.0.0, we’re encountering compatibility errors with system indices (primarily .kibana_* indices) that were originally created in Elasticsearch 7.10.2 but have been upgraded to OpenSearch 2.19.0 at some point in the past:

The index [[.kibana_xxx/uuid]] was created with version [7.10.2] but the minimum compatible version is [2.0.0]. It should be re-indexed in OpenSearch 2.x before upgrading to 3.0.0.

Checking the _settings?human for that index returns the following:

        "version": {
          "upgraded": "136407827",
          "created_string": "7.10.2",
          "upgraded_string": "2.19.0",
          "created": "7100299"
        }

Is it expected behavior that system indices (.kibana_*, etc.) that were originally created in Elasticsearch 7.x (but upgraded since) would cause compatibility issues during an upgrade? These indices contain configuration data and are usually managed by OpenSearch/Kibana.

The upgrade documentation index compatibility reference ends in OpenSearch versions 2.13.0/2.14.0 which suggests that anything above that would likely be compatible with 3.0.0, or is this a misinterpretation?

The indices function correctly in OpenSearch 2.19.2 and contain the upgraded_string: "2.19.0" field.

I’m still in the process of developing a workaround, but so far it has involved reindexing the indices to a new temporary index with the same template, deleting the original index, creating a new one with the same original name and template, and reindexing from the temporary index back to the new one. Do be aware that this is just a proof of concept, but doing so seems to not have broken any functionality, and the new index settings now shows the following:

        "version": {
          "created_string": "2.19.2",
          "created": "136408027"
        }

… which I’m hopeful will fix the issue.

However, two questions remain:

  • Is this a bug or expected behavior?
  • Are system indexes also supposed to be reindexed to match the compatibility reference table constraints?

Thanks.

2

I am facing similar issue

opensearch Caused by: java.lang.IllegalStateException: The index [[.opendistro-anomaly-detection-state/VB38MmyG-HSKDug]] was created with version [1.1.0] but the minimum compatible version is [2.0.0]. It should be re-indexed in OpenSearch 2.x before upgrading to 3.1.0

This seems to be very old index created during creation of this cluster

{
  ".opendistro-anomaly-detection-state": {
    "settings": {
      "index": {
        "creation_date_string": "2021-11-30T09:14:58.621Z",
        "hidden": "true",
        "number_of_shards": "1",
        "provided_name": ".opendistro-anomaly-detection-state",
        "creation_date": "1638263698621",
        "number_of_replicas": "1",
        "uuid": "VB38MmyG-HSKDug",
        "version": {
          "upgraded": "136327927",
          "created_string": "1.1.0",
          "upgraded_string": "2.11.1",
          "created": "135227827"
        }
      }
    }
  }
}

I had to abort the upgrade and revert back. Any solutions will be appreciated.