Upgrade 1.36 -> 2.0 + alerting

OpenSearch
, ,
1

Hi everyone, I am attempting an upgrade from 1.36 → 2.x.
When starting up the nodes after a full cluster shutdown, upgrade etc. quite a few .opendistro* indices had an error of:

org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: The index [[.opendistro-alerting-alerts/xOoMmmnISrS1VwwOpFEdvA]] was created with version [6.6.2] but the minimum compatible version is OpenSearch 1.0.0 (or Elasticsearch 7.0.0). It should be re-indexed in OpenSearch 1.x (or Elasticsearch 7.x) before upgrading to 2.0.1.

When trying to re-index I see:

curl -k --cert ./admin.pem --key ./admin-key.pem -XPOST 'https://localhost:9200/_reindex' -H "Content-type:application/json" -d '{"source":{"index":".opendistro-alerting-config"}, "dest":{"index":"reindexed.opendistro-alerting-config"}}'
{"took":6051,"timed_out":false,"total":339,"updated":0,"created":252,"deleted":0,"batches":1,"version_conflicts":0,"noops":0,"retries":{"bulk":0,"search":0},"throttled_millis":0,"requests_per_second":-1.0,"throttled_until_millis":0,"failures":[{"index":"reindexed.opendistro-alerting-config","type":"_doc","id":"HDTI6nMBC68yrcbDBG5F","cause":{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] has been exceeded"},"status":400}
...

{ - 
  ".opendistro-alerting-config": { - 
    "settings": { - 
      "index": { - 
        "creation_date": "1556302115948",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "4EjribS7R5CdwYdYxzE09g",
        "version": { - 
          "created": "6060299",
          "upgraded": "135248427"
        },
        "provided_name": ".opendistro-alerting-config"
      }
    }
  }
}

The index settings include “upgraded” but that doesnt seem to help.

I guess my question is has anyone successfully upgraded while preserving the alerting config index? or any of the other system indices that seem to be problematic?

edit
I read through https://forum.opensearch.org/t/impossible-upgrade-opensearch-1-3-2-to-2-0-0-index-version-incompatible but there didnt seem to be any specifics around the alerting indices, only the .kibana indices.

2

Ack! Were you able to get through this @jasonrojas ?

@nateynate @dtaivpp - any ideas?

3

I did. basically I did something to the effect of:

curl -k --cert ./admin.pem --key ./admin-key.pem -XPOST 'https://localhost:9200/_reindex?wait_for_completion=true' -H "Content-type:application/json" -d '{"source":{"index":".opendistro-alerting-config"}, "dest":{"index":"reindexed.opendistro-alerting-config"}}'

curl -k --cert ./admin.pem --key ./admin-key.pem -XDELETE 'https://localhost:9200/.opendistro-alerting-config'

curl -k --cert ./admin.pem --key ./admin-key.pem -XPOST 'https://localhost:9200/_reindex?wait_for_completion=true' -H "Content-type:application/json" -d '{"source":{"index":"reindexed.opendistro-alerting-config"}, "dest":{"index":".opendistro-alerting-config"}}'

I might have had to do some extra work with exporting the mapping to and creating an index template for it so ymmv.

4 Likes
4

thanks @jasonrojas - other community members will benefit from this info

1 Like