Reindex system indices?

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): 3.2.0

Describe the issue: We are experiencing issues with indices that have enabled segment replication, see issue [BUG] Shard fails to re-assign after a rolling restart · Issue #19234 · opensearch-project/OpenSearch · GitHub .

We previously enabled segment replication cluster wide and have since disabled this, but older indices and all system indices have segment replication enabled. For example:

{
  ".opensearch-sap--job": {
    "settings": {
      "index": {
        "replication": {
          "type": "SEGMENT"
        },
        "hidden": "true",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-all",
        "provided_name": ".opensearch-sap--job",
        "creation_date": "1748899041215",
        "number_of_replicas": "5",
        "uuid": "gXsejkm4SP-n5rZt2vMvxQ",
        "version": {
          "created": "137217827"
        }
      }
    }
  }
}

As far as i can tell, we can’t disable or “switch back” to DOCUMENT replication for these indices. So i was thinking if we could reindex these system indices? Is it possible to switch system indices to a reindex copy of said index?

@victor You should be able to reindex, first to temporary index, and then back to originally named index. I would recommend to block writes to the indices in the mean time, using:

PUT /.opensearch-sap--job/_settings
{
  "index": { "blocks.write": true }
}

Also note that some of the system indices will require admin certificate and key to be reindexed.